Nov 02 06:13:34.146682 ip-10-0-130-34 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 02 06:13:34.146695 ip-10-0-130-34 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 02 06:13:34.146703 ip-10-0-130-34 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 02 06:13:34.146990 ip-10-0-130-34 systemd[1]: Failed to start Kubernetes Kubelet. Nov 02 06:13:44.185747 ip-10-0-130-34 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 02 06:13:44.185761 ip-10-0-130-34 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 93ac4e0a77f84e3595d81698722f20e3 -- Nov 02 06:15:50.098399 ip-10-0-130-34 systemd[1]: Starting Kubernetes Kubelet... Nov 02 06:15:50.507220 ip-10-0-130-34 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:15:50.507220 ip-10-0-130-34 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 06:15:50.507220 ip-10-0-130-34 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:15:50.507220 ip-10-0-130-34 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:15:50.512175 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512070 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:50.512175 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512170 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:50.512175 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512176 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:50.512175 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512181 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:50.512175 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512185 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512189 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512193 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512197 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512201 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512204 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512208 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512213 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512216 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512220 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512224 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512227 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512232 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512236 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512240 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512243 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512247 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512251 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512255 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512259 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:50.512475 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512264 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512268 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512272 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512276 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512280 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512284 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512287 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512291 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512296 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512300 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512304 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512308 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512313 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512317 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512323 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512327 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512334 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512340 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512347 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512351 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:50.513313 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512356 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512360 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512364 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512369 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512373 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512377 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512381 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512386 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512390 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512394 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512398 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512404 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512408 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512412 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512416 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512421 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512425 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512430 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512434 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:50.514127 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512438 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512443 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512447 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512461 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512465 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512469 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512473 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512478 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512482 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512486 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512490 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512495 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512499 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512503 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512508 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512512 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512516 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512520 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512525 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512529 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:50.514928 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512534 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512538 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512542 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512546 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512550 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512555 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512560 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512565 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512569 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512580 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512587 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512594 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512599 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512604 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512609 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512613 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512618 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512622 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512626 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:50.515799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512631 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512638 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512642 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.512646 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.513956 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.513966 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.513972 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.513977 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.513981 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.513985 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.513989 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.513994 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.513999 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514003 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514008 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514012 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514016 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514020 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514025 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514030 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:50.516328 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514034 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514039 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514043 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514047 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514051 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514056 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514061 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514065 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514070 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514074 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514078 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514082 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514087 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514092 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514096 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514100 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514105 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514109 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514113 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:50.516822 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514117 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514122 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514125 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514153 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514157 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514162 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514166 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514170 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514174 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514178 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514182 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514186 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514190 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514196 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514200 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514204 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514209 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514213 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514217 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514222 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:50.517292 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514226 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514231 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514235 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514240 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514244 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514248 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514253 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514257 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514263 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514267 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514271 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514276 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514280 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514284 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514288 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514292 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514296 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514300 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514304 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514309 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:50.517766 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514313 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514317 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514322 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514328 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514334 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514339 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514345 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514350 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514354 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514358 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514362 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514367 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514371 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514378 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514383 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514387 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514392 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514397 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514402 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:50.518357 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514406 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514410 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514414 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514419 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514423 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514427 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514431 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514435 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514441 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514446 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514450 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.514454 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515291 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515300 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515305 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515310 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515314 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515318 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515322 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:50.518815 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515328 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515332 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515336 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515341 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515345 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515350 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515354 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515358 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515362 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515367 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515371 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515376 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515380 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515385 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515389 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515393 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515398 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515402 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515406 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515410 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:50.519289 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515415 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515419 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515423 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515427 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515431 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515436 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515440 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515445 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515449 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515453 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515458 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515462 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515466 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515472 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515476 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515480 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515484 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515489 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515493 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515497 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:50.519771 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515501 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515505 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515509 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515515 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515519 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515523 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515527 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515531 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515535 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515539 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515543 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515547 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515552 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515556 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515560 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515564 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515568 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515573 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515577 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515581 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:50.520256 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515585 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515589 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515593 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515598 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515602 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515607 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515612 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515616 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515633 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515642 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515646 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515650 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515655 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515660 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515664 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515669 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515673 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515677 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515682 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515686 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:50.520734 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515691 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515695 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515700 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515704 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515708 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515712 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515717 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515721 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515725 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515729 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515733 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515740 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515748 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515755 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515759 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515764 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515768 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515772 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.515776 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517382 2574 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 06:15:50.521275 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517400 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517407 2574 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517413 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517418 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517421 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517426 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517431 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517434 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517437 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517441 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517445 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517448 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517451 2574 flags.go:64] FLAG: --cgroup-root="" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517454 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517458 2574 flags.go:64] FLAG: --client-ca-file="" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517482 2574 flags.go:64] FLAG: --cloud-provider="external" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517487 2574 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517491 2574 flags.go:64] FLAG: --cluster-domain="" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517494 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517498 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517501 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517504 2574 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517508 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 06:15:50.521745 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517511 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517515 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517518 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517521 2574 flags.go:64] FLAG: --contention-profiling="false" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517524 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517528 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517531 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517534 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517539 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517544 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517547 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517550 2574 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517554 2574 flags.go:64] FLAG: --enable-server="true" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517557 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517562 2574 flags.go:64] FLAG: --event-burst="100" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517565 2574 flags.go:64] FLAG: --event-qps="50" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517568 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517572 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517575 2574 flags.go:64] FLAG: --eviction-hard="" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517580 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517583 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517586 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517589 2574 flags.go:64] FLAG: --eviction-soft="" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517593 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517596 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 06:15:50.522311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517599 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517602 2574 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517606 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517609 2574 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517612 2574 flags.go:64] FLAG: --feature-gates="" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517616 2574 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517619 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517622 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517626 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517629 2574 flags.go:64] FLAG: --healthz-port="10248" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517632 2574 flags.go:64] FLAG: --help="false" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517635 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517639 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517642 2574 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517645 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517650 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517654 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517657 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517661 2574 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517664 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517667 2574 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517670 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517673 2574 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517676 2574 flags.go:64] FLAG: --kube-reserved="" Nov 02 06:15:50.522910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517679 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517682 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517685 2574 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517688 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517691 2574 flags.go:64] FLAG: --lock-file="" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517694 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517697 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517700 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517706 2574 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517709 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517712 2574 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517715 2574 flags.go:64] FLAG: --logging-format="text" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517718 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517721 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517724 2574 flags.go:64] FLAG: --manifest-url="" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517728 2574 flags.go:64] FLAG: --manifest-url-header="" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517733 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517736 2574 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517740 2574 flags.go:64] FLAG: --max-pods="110" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517743 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517746 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517749 2574 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517753 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517757 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517761 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 06:15:50.523495 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517765 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517776 2574 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517779 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517784 2574 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517787 2574 flags.go:64] FLAG: --pod-cidr="" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517790 2574 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517793 2574 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517796 2574 flags.go:64] FLAG: --pods-per-core="0" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517799 2574 flags.go:64] FLAG: --port="10250" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517802 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517805 2574 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f3cf3049f5a29920" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517808 2574 flags.go:64] FLAG: --qos-reserved="" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517811 2574 flags.go:64] FLAG: --read-only-port="10255" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517814 2574 flags.go:64] FLAG: --register-node="true" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517817 2574 flags.go:64] FLAG: --register-with-taints="" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517822 2574 flags.go:64] FLAG: --registry-burst="10" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517825 2574 flags.go:64] FLAG: --registry-qps="5" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517828 2574 flags.go:64] FLAG: --reserved-cpus="" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517831 2574 flags.go:64] FLAG: --reserved-memory="" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517835 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517838 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517841 2574 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517844 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517847 2574 flags.go:64] FLAG: --runonce="false" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517850 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517853 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517856 2574 flags.go:64] FLAG: --seccomp-default="false" Nov 02 06:15:50.524101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517860 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517863 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517867 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517870 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517874 2574 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517877 2574 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517880 2574 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517883 2574 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517886 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517889 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517892 2574 flags.go:64] FLAG: --system-cgroups="" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517895 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517902 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517905 2574 flags.go:64] FLAG: --tls-cert-file="" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517908 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517912 2574 flags.go:64] FLAG: --tls-min-version="" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517915 2574 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517917 2574 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517920 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517923 2574 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517926 2574 flags.go:64] FLAG: --v="2" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517930 2574 flags.go:64] FLAG: --version="false" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517937 2574 flags.go:64] FLAG: --vmodule="" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517942 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.517945 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 06:15:50.524799 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518089 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518093 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518096 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518099 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518102 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518105 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518108 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518111 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518113 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518116 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518119 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518123 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518126 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518142 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518145 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518152 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518154 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518157 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518160 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:50.525402 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518163 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518165 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518168 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518170 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518173 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518175 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518178 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518181 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518184 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518186 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518189 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518192 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518194 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518197 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518199 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518202 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518204 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518207 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518209 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518212 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:50.525860 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518214 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518217 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518221 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518224 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518228 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518230 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518233 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518235 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518238 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518241 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518244 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518246 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518248 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518251 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518253 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518256 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518258 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518261 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518263 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:50.526590 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518265 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518268 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518270 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518273 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518275 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518278 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518280 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518283 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518286 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518288 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518291 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518293 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518296 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518298 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518301 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518303 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518306 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518308 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518311 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518313 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:50.527213 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518316 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518318 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518321 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518324 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518327 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518329 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518332 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518334 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518337 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518339 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518342 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518344 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518347 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518350 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518352 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518355 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518357 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518361 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518366 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:50.527704 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518373 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518376 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518379 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518382 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518385 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518388 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518391 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518394 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518397 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.518426 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518468 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518473 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518475 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518478 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518481 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518484 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:50.528199 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518488 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518491 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518493 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518496 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518499 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518501 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518505 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518509 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518512 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518515 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518518 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518521 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518537 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518542 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518545 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518548 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518551 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518555 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518558 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:50.528607 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518561 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518564 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518566 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518568 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518571 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518574 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518576 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518581 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518584 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518587 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518589 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518592 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518595 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518598 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518601 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518604 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518607 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518610 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518613 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:50.529063 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518615 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518618 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518620 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518623 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518625 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518628 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518630 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518633 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518636 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518638 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518641 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518645 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518647 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518650 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518652 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518655 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518657 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518660 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518663 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:50.529529 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518665 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518670 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518674 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518677 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518680 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518682 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518685 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518687 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518690 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518693 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518696 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518698 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518701 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518703 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518706 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518708 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518711 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518713 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518716 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:50.529976 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518718 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518721 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518724 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518726 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518728 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518733 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518737 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518739 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518742 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518745 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518748 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518750 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518753 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518755 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518760 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518762 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518765 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518767 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518770 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518772 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:50.530480 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518775 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518778 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518780 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.518783 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.525695 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.525711 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.525728 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.525733 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525786 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525792 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525795 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525798 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525801 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525804 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525806 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525809 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525812 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525815 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525817 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:50.530966 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525820 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525823 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525825 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525828 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525830 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525833 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525836 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525838 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525841 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525844 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525846 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525850 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525853 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525856 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525858 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525861 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525865 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525867 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.525914 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526121 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:50.531433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526129 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526149 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526154 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526159 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526167 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526173 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526180 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526185 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526189 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526197 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526201 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526206 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526216 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526220 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526224 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526228 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526235 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526241 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526245 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:50.531904 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526250 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526255 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526259 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526263 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526268 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526280 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526285 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526290 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526294 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526298 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526302 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526306 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526312 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526316 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526320 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526324 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526328 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526333 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526341 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526345 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:50.532369 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526350 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526354 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526359 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526363 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526367 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526371 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526375 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526379 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526384 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526388 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526392 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526401 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526405 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526410 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526414 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526419 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526424 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526428 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526433 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:50.532855 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526438 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526442 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526446 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526450 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526459 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526463 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526468 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526472 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526476 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526480 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526484 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526489 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526493 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526497 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526501 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526505 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:50.533321 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.526509 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.526563 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527322 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527339 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527345 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527349 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527354 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527358 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527362 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527366 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527370 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527375 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527380 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527384 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527389 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527393 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:50.533702 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527399 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527404 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527408 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527412 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527416 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527421 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527426 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527430 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527434 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527438 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527442 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527446 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527450 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527454 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527459 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527463 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527467 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527471 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527476 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527480 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527485 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:50.534120 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527489 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527494 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527498 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527502 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527506 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527512 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527516 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527520 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527524 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527529 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527535 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527539 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527543 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527546 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527549 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527552 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527555 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527558 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527561 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:50.534750 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527564 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527566 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527569 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527572 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527575 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527577 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527580 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527583 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527585 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527588 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527591 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527594 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527597 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527599 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527602 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527605 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527608 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527612 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527614 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:50.535281 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527617 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527619 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527622 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527625 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527627 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527630 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527632 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527635 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527639 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527643 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527646 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527650 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527653 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527656 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527659 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527662 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527665 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527667 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527670 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:50.535738 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527673 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527676 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527678 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527681 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527684 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527688 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527690 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527693 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527696 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527699 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527702 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527705 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527708 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527711 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.527747 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:15:50.536210 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527786 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527790 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527793 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527796 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527798 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527801 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527804 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527807 2574 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527809 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527812 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527815 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527818 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527820 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527823 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527825 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527828 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527830 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527833 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527835 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:15:50.536565 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527838 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527841 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527843 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527847 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527849 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527852 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527855 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527857 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527860 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527863 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527865 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527869 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527873 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527877 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527879 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527882 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527885 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527887 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527890 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:15:50.537059 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527892 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527895 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527897 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527900 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527903 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527905 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527908 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527911 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527913 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527916 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527918 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527921 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527924 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527926 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527929 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527931 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527935 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527937 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527940 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:15:50.537626 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527942 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527945 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527947 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527950 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527952 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527955 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527958 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527961 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527965 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527968 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527970 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527973 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527975 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527978 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527981 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527983 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527986 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527988 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527991 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527993 2574 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:15:50.538092 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527996 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.527999 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528001 2574 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528004 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528006 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528009 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528012 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528014 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528017 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528020 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528023 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528025 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528028 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528030 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528033 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528035 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528038 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528041 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528044 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528047 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:15:50.538612 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528049 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528052 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528055 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528059 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528062 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528066 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528069 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528073 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:50.528075 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.528154 2574 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.530899 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.531809 2574 server.go:1014] "Starting client certificate rotation" Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.531904 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.531939 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.533887 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:15:50.539225 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.535040 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:15:50.544705 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.544689 2574 log.go:25] "Validated CRI v1 runtime API" Nov 02 06:15:50.547753 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.547740 2574 log.go:25] "Validated CRI v1 image API" Nov 02 06:15:50.548610 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.548580 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 06:15:50.555243 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.555217 2574 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8397d13b-509b-48f6-b958-46b99041e4e4:/dev/nvme0n1p4 d7685f30-41ba-435d-bd41-0585f999f490:/dev/nvme0n1p3] Nov 02 06:15:50.555310 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.555242 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 02 06:15:50.560358 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.560248 2574 manager.go:217] Machine: {Timestamp:2026-11-02 06:15:50.558988415 +0000 UTC m=+0.341742967 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102349 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b39ea8dcee5b2e8ff8e6249a4b871 SystemUUID:ec2b39ea-8dce-e5b2-e8ff-8e6249a4b871 BootID:93ac4e0a-77f8-4e35-95d8-1698722f20e3 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:e3:0c:b7:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:e3:0c:b7:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:1a:a5:9c:c4:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 06:15:50.560358 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.560358 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 06:15:50.560486 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.560474 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 06:15:50.562970 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.562942 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 06:15:50.563105 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.562973 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-34.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 06:15:50.563157 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.563107 2574 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 06:15:50.563157 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.563115 2574 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 06:15:50.563157 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.563147 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:15:50.563273 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.563162 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:15:50.564015 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.564005 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 06:15:50.567548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.567536 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 06:15:50.567664 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.567655 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 06:15:50.568338 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.568329 2574 kubelet.go:493] "Attempting to sync node with API server" Nov 02 06:15:50.568367 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.568346 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 06:15:50.568367 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.568355 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 06:15:50.568367 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.568364 2574 kubelet.go:405] "Adding apiserver pod source" Nov 02 06:15:50.568458 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.568371 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 06:15:50.568840 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.568826 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:15:50.573208 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.573089 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 06:15:50.577949 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.577929 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:15:50.578692 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.578680 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 06:15:50.578730 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.578710 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 06:15:50.578862 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.578852 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 06:15:50.578890 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.578867 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 06:15:50.578890 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.578874 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 06:15:50.578890 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.578879 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 06:15:50.578890 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.578886 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 06:15:50.578890 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.578891 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 06:15:50.579587 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.579578 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 06:15:50.579619 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.579588 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 06:15:50.579619 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.579596 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 06:15:50.579619 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.579602 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 06:15:50.579619 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.579611 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 06:15:50.579619 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.579619 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 06:15:50.580363 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.580355 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 06:15:50.580400 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.580364 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 06:15:50.587862 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.587850 2574 server.go:1263] "Started kubelet" Nov 02 06:15:50.587978 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.587950 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 06:15:50.588019 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.587947 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 06:15:50.588071 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.588027 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 06:15:50.588672 ip-10-0-130-34 systemd[1]: Started Kubernetes Kubelet. Nov 02 06:15:50.589620 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.589567 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 06:15:50.589704 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.589666 2574 server.go:317] "Adding debug handlers to kubelet server" Nov 02 06:15:50.595481 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.595315 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 06:15:50.596333 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.596315 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 06:15:50.596761 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:50.596741 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-34.us-west-2.compute.internal\" not found" Nov 02 06:15:50.597010 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.596994 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 06:15:50.597147 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:50.597106 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 02 06:15:50.597249 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.597168 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 06:15:50.597249 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.597180 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 06:15:50.597249 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.597205 2574 factory.go:55] Registering systemd factory Nov 02 06:15:50.597249 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.597222 2574 factory.go:223] Registration of the systemd container factory successfully Nov 02 06:15:50.597408 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.597336 2574 reconstruct.go:97] "Volume reconstruction finished" Nov 02 06:15:50.597408 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.597344 2574 reconciler.go:29] "Reconciler: start to sync state" Nov 02 06:15:50.597482 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.597434 2574 factory.go:153] Registering CRI-O factory Nov 02 06:15:50.597482 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.597449 2574 factory.go:223] Registration of the crio container factory successfully Nov 02 06:15:50.597551 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.597491 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 06:15:50.597551 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.597510 2574 factory.go:103] Registering Raw factory Nov 02 06:15:50.597551 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.597523 2574 manager.go:1199] Started watching for new ooms in manager Nov 02 06:15:50.598523 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.598511 2574 manager.go:319] Starting recovery of all containers Nov 02 06:15:50.610669 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.608609 2574 manager.go:324] Recovery completed Nov 02 06:15:50.614067 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.614051 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:50.615732 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.615714 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:50.615802 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.615745 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:50.615802 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.615756 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:50.616325 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.616312 2574 cpu_manager.go:226] "Starting" policy="none" Nov 02 06:15:50.616325 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.616324 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 06:15:50.616442 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.616344 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 06:15:50.616442 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.616367 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:50.617162 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.617112 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:50.617162 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.617159 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:50.617280 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.617175 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:50.618755 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.618423 2574 policy_none.go:50] "Start" Nov 02 06:15:50.618755 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.618435 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 06:15:50.618755 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.618445 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 06:15:50.620318 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.620304 2574 policy_none.go:44] "Start" Nov 02 06:15:50.659813 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.659778 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 06:15:50.663047 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.663031 2574 manager.go:341] "Starting Device Plugin manager" Nov 02 06:15:50.663149 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:50.663065 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 06:15:50.663149 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.663078 2574 server.go:87] "Starting device plugin registration server" Nov 02 06:15:50.663354 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.663342 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 06:15:50.663399 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.663357 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 06:15:50.663456 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.663437 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 06:15:50.663575 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.663559 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 06:15:50.663619 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.663575 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 06:15:50.664154 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:50.664116 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 06:15:50.664240 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:50.664201 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-34.us-west-2.compute.internal\" not found" Nov 02 06:15:50.730573 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.730539 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 06:15:50.732561 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.731793 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 06:15:50.732561 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:50.730612 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-34.us-west-2.compute.internal.18e425d584dfce56 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-34.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-34.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 06:15:50.587821654 +0000 UTC m=+0.370576195,LastTimestamp:2026-11-02 06:15:50.587821654 +0000 UTC m=+0.370576195,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-34.us-west-2.compute.internal,}" Nov 02 06:15:50.732561 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.732233 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 06:15:50.732561 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.732270 2574 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 06:15:50.732561 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:50.732313 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 02 06:15:50.765260 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.765164 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:50.765943 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.765929 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:50.766005 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.765958 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:50.766005 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.765968 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:50.766005 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.765992 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.780993 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.780966 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-34.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 02 06:15:50.781100 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:50.781069 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-34.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 02 06:15:50.821025 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.820997 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 06:15:50.821148 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.821042 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6d8jp" Nov 02 06:15:50.832671 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.832651 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal"] Nov 02 06:15:50.832726 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.832712 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:50.833727 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.833712 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:50.833811 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.833748 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:50.833811 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.833765 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:50.836007 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.835992 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:50.836637 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.836621 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.836726 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.836652 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:50.836787 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.836735 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:50.836787 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.836766 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:50.836787 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.836781 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:50.837347 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.837333 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:50.837422 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.837357 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:50.837422 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.837368 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:50.838876 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.838859 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.838962 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.838889 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:15:50.840478 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.840464 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:15:50.840571 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.840492 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:15:50.840571 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.840506 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:15:50.866585 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:50.866570 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-34.us-west-2.compute.internal\" not found" node="ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.869654 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:50.869637 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-34.us-west-2.compute.internal\" not found" node="ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.880789 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.880774 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:50.882315 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.882300 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:50.898633 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.898612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceea1ff964927afa3d04bca3afe81300-config\") pod \"kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"ceea1ff964927afa3d04bca3afe81300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.898709 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.898637 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.898709 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.898654 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.972904 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.972879 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:50.974575 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.974558 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:50.975260 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.975244 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.975320 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:50.975266 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-34.us-west-2.compute.internal\": node \"ip-10-0-130-34.us-west-2.compute.internal\" not found" Nov 02 06:15:50.997607 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.997579 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.999015 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.998997 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceea1ff964927afa3d04bca3afe81300-config\") pod \"kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"ceea1ff964927afa3d04bca3afe81300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.999103 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.999021 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.999103 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.999038 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.999103 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.999057 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceea1ff964927afa3d04bca3afe81300-config\") pod \"kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"ceea1ff964927afa3d04bca3afe81300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.999103 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.999093 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:50.999103 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:50.999101 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:51.054092 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:51.054039 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:15:51.054181 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:51.054159 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:51.107112 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:51.107091 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:15:51.170743 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:51.170725 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:51.173769 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:51.173747 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:15:52.559986 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.559952 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 02 06:15:52.560496 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.560106 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 06:15:52.560496 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.560121 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 06:15:52.560496 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.560171 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 02 06:15:52.560496 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:52.560173 2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://af4ae981bd91148c598c3a76754e16e4-db7866749802e67b.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-34.us-west-2.compute.internal.18e425d58689aef9\": read tcp 10.0.130.34:45586->18.217.150.161:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-34.us-west-2.compute.internal.18e425d58689aef9 default 4903 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-34.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-34.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-34.us-west-2.compute.internal,},FirstTimestamp:2026-11-02 06:15:50 +0000 UTC,LastTimestamp:2026-11-02 06:15:50.833727211 +0000 UTC m=+0.616481753,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-34.us-west-2.compute.internal,}" Nov 02 06:15:52.596964 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.596946 2574 apiserver.go:52] "Watching apiserver" Nov 02 06:15:52.623842 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.623821 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 02 06:15:52.844187 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.844117 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 06:15:52.844453 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.844431 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-gm9nl","openshift-multus/multus-c9z9s","openshift-network-operator/iptables-alerter-8tczj","openshift-ovn-kubernetes/ovnkube-node-x4qlj","kube-system/konnectivity-agent-fc9xt","openshift-dns/node-resolver-5jr5c","kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal","openshift-multus/network-metrics-daemon-5wc8x","openshift-network-diagnostics/network-check-target-k7c7h","kube-system/global-pull-secret-syncer-4vv5l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc","openshift-cluster-node-tuning-operator/tuned-ccb2x","openshift-image-registry/node-ca-58ckp"] Nov 02 06:15:52.846835 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.846818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:15:52.848752 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.848737 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.850689 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.850676 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.852695 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.852680 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:15:52.854130 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.854103 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 06:10:51 +0000 UTC" deadline="2028-06-30 22:31:05.34567228 +0000 UTC" Nov 02 06:15:52.854185 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.854131 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14560h15m12.491544951s" Nov 02 06:15:52.854780 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.854767 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.856793 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.856775 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:15:52.858797 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.858782 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:52.858863 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:52.858839 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:15:52.860985 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.860970 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:15:52.863266 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.863249 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:52.865396 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.865379 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:15:52.865484 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:52.865429 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:15:52.867644 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.867624 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:52.869786 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.869768 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:52.869870 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:52.869830 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:15:52.895234 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.895215 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 06:15:52.895234 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.895215 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d57td\"" Nov 02 06:15:52.895364 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.895241 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 06:15:52.896473 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.896449 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 06:15:52.896560 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.896489 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:15:52.896560 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.896537 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xffwp\"" Nov 02 06:15:52.898609 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.898594 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 06:15:52.898609 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.898603 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 06:15:52.898727 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.898639 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 06:15:52.898813 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.898794 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bpw78\"" Nov 02 06:15:52.898916 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.898902 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 06:15:52.900398 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.900380 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:15:52.900658 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.900639 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 06:15:52.900760 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.900702 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 06:15:52.900822 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.900807 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77l82\"" Nov 02 06:15:52.902584 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.902568 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-flhk8\"" Nov 02 06:15:52.905004 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.902980 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 06:15:52.925896 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.925880 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 06:15:52.931715 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.931694 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:52.931790 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.931724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw9sf\" (UniqueName: \"kubernetes.io/projected/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-kube-api-access-sw9sf\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.931790 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.931739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p667p\" (UniqueName: \"kubernetes.io/projected/33574457-8ee3-4f93-80b7-3e961da45892-kube-api-access-p667p\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:52.931790 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.931761 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-cnibin\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:52.931927 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.931815 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-ovn\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.931927 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.931862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-k8s-cni-cncf-io\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.932000 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.931917 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-konnectivity-ca\") pod \"konnectivity-agent-fc9xt\" (UID: \"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad\") " pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:15:52.932000 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.931988 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1edfb6b-cbff-47e6-aab2-a942731747b8-hosts-file\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:15:52.932070 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932005 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-modprobe-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.932070 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932024 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-kubernetes\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.932070 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932042 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-kubelet\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.932180 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932086 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-device-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:52.932180 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932131 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-sys-fs\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:52.932180 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932157 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.932284 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932184 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-socket-dir-parent\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.932284 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghc4z\" (UniqueName: \"kubernetes.io/projected/08d074a5-dadb-4e71-a67f-9bc5c33d8857-kube-api-access-ghc4z\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:15:52.932284 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-etc-tuned\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.932284 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932263 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-multus\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.932451 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-multus-certs\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.932451 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932367 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:52.932451 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932414 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:52.932575 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932455 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-registration-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:52.932575 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932488 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.932575 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932524 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08d074a5-dadb-4e71-a67f-9bc5c33d8857-iptables-alerter-script\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:15:52.932575 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.932723 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932590 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70e61026-bb17-490c-aed0-c7387729c9ef-serviceca\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:15:52.932723 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932618 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-agent-certs\") pod \"konnectivity-agent-fc9xt\" (UID: \"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad\") " pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:15:52.932723 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:52.932723 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-etc-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.932723 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932680 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovn-node-metrics-cert\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.932723 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932717 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-dbus\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:52.932956 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-node-log\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.932956 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-config\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.932956 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932782 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysconfig\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.932956 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932812 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-conf\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.932956 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-kubelet\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.932956 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-slash\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.932956 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-env-overrides\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.932956 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932936 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-kubelet-config\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:52.933309 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932960 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-netd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.933309 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.932998 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:52.933309 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933044 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:52.933309 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-systemd-units\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.933309 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-systemd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.933309 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933140 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zc6b\" (UniqueName: \"kubernetes.io/projected/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-kube-api-access-9zc6b\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.933309 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933167 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dkw\" (UniqueName: \"kubernetes.io/projected/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kube-api-access-24dkw\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:52.933309 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933194 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-netns\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.933309 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933215 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.933309 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933256 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933327 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-bin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933358 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-conf-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933382 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-netns\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933415 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-var-lib-kubelet\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cnibin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933485 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tsjq\" (UniqueName: \"kubernetes.io/projected/aed3a242-64df-41a6-98b3-e2a6543918fb-kube-api-access-6tsjq\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933502 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70e61026-bb17-490c-aed0-c7387729c9ef-host\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933520 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkwvv\" (UniqueName: \"kubernetes.io/projected/70e61026-bb17-490c-aed0-c7387729c9ef-kube-api-access-jkwvv\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933537 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zljkr\" (UniqueName: \"kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr\") pod \"network-check-target-k7c7h\" (UID: \"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef\") " pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-socket-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933578 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-systemd\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933592 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-lib-modules\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.933661 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933641 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96tbp\" (UniqueName: \"kubernetes.io/projected/513b68de-5cb3-4038-9592-5eb8989af256-kube-api-access-96tbp\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933669 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-system-cni-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-var-lib-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-run\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-sys\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-host\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-tmp\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933867 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-system-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-hostroot\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933923 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-log-socket\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.933992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1edfb6b-cbff-47e6-aab2-a942731747b8-tmp-dir\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.934047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw58c\" (UniqueName: \"kubernetes.io/projected/e1edfb6b-cbff-47e6-aab2-a942731747b8-kube-api-access-dw58c\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.934132 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cni-binary-copy\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.934170 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-daemon-config\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.934233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.934218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-etc-kubernetes\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:52.934669 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.934263 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08d074a5-dadb-4e71-a67f-9bc5c33d8857-host-slash\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:15:52.934669 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.934294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-bin\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.934669 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.934325 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-script-lib\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:52.938387 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.938372 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 06:15:52.938527 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.938513 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 06:15:52.941029 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.941016 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 06:15:52.941095 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.941045 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 06:15:52.941137 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.941090 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 06:15:52.992488 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.992467 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 02 06:15:52.992610 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.992532 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xf5qw" Nov 02 06:15:52.995463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995439 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 06:15:52.995463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995455 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 06:15:52.995639 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995492 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rwgrc\"" Nov 02 06:15:52.995639 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995506 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 06:15:52.995639 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995537 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zth7z\"" Nov 02 06:15:52.995639 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995568 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 06:15:52.995639 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995582 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 06:15:52.995639 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995611 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 06:15:52.995639 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995616 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 06:15:52.995639 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995617 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d5zsv\"" Nov 02 06:15:52.995639 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995617 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 06:15:52.995639 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995631 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 06:15:52.995914 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995645 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 06:15:52.995914 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:52.995659 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4lqj8\"" Nov 02 06:15:53.034908 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.034880 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-netns\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.034908 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.034911 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.035146 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.034928 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-var-lib-kubelet\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.035146 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.034956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cnibin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.035146 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.034980 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tsjq\" (UniqueName: \"kubernetes.io/projected/aed3a242-64df-41a6-98b3-e2a6543918fb-kube-api-access-6tsjq\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.035146 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.034991 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-netns\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.035146 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035004 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70e61026-bb17-490c-aed0-c7387729c9ef-host\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:15:53.035146 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035012 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.035146 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035035 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-var-lib-kubelet\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.035146 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035041 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwvv\" (UniqueName: \"kubernetes.io/projected/70e61026-bb17-490c-aed0-c7387729c9ef-kube-api-access-jkwvv\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:15:53.035146 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035057 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70e61026-bb17-490c-aed0-c7387729c9ef-host\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:15:53.035146 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035041 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cnibin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljkr\" (UniqueName: \"kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr\") pod \"network-check-target-k7c7h\" (UID: \"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef\") " pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-socket-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035274 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-systemd\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-lib-modules\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035324 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tbp\" (UniqueName: \"kubernetes.io/projected/513b68de-5cb3-4038-9592-5eb8989af256-kube-api-access-96tbp\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035338 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-socket-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035347 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-system-cni-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035352 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-systemd\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035371 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-var-lib-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035393 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-run\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035408 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-system-cni-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035409 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-var-lib-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035414 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-sys\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035442 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-run\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035449 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-lib-modules\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035456 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-sys\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.035463 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035459 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-host\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035501 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-host\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-tmp\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-system-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035574 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-hostroot\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-log-socket\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035606 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035627 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1edfb6b-cbff-47e6-aab2-a942731747b8-tmp-dir\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035641 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw58c\" (UniqueName: \"kubernetes.io/projected/e1edfb6b-cbff-47e6-aab2-a942731747b8-kube-api-access-dw58c\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035649 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-system-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035648 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-hostroot\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035657 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cni-binary-copy\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035679 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-daemon-config\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035684 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-log-socket\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-etc-kubernetes\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08d074a5-dadb-4e71-a67f-9bc5c33d8857-host-slash\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035750 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035779 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-bin\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.036197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-script-lib\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035835 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-bin\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035838 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035857 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035877 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw9sf\" (UniqueName: \"kubernetes.io/projected/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-kube-api-access-sw9sf\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035909 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p667p\" (UniqueName: \"kubernetes.io/projected/33574457-8ee3-4f93-80b7-3e961da45892-kube-api-access-p667p\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.035934 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035952 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08d074a5-dadb-4e71-a67f-9bc5c33d8857-host-slash\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035980 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-cnibin\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.036016 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret podName:6ab6077a-6df9-4de9-8632-98830b9a8653 nodeName:}" failed. No retries permitted until 2026-11-02 06:15:53.53597676 +0000 UTC m=+2.290788776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret") pod "global-pull-secret-syncer-4vv5l" (UID: "6ab6077a-6df9-4de9-8632-98830b9a8653") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035935 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-cnibin\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-ovn\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-k8s-cni-cncf-io\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-konnectivity-ca\") pod \"konnectivity-agent-fc9xt\" (UID: \"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad\") " pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1edfb6b-cbff-47e6-aab2-a942731747b8-hosts-file\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-modprobe-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035910 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-etc-kubernetes\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.037038 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-kubernetes\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-kubelet\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036236 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-k8s-cni-cncf-io\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-device-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036279 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-ovn\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036279 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-sys-fs\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036337 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-sys-fs\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-socket-dir-parent\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036343 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-modprobe-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036369 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghc4z\" (UniqueName: \"kubernetes.io/projected/08d074a5-dadb-4e71-a67f-9bc5c33d8857-kube-api-access-ghc4z\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036382 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-socket-dir-parent\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036383 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-daemon-config\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-etc-tuned\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-multus\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036393 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cni-binary-copy\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036446 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1edfb6b-cbff-47e6-aab2-a942731747b8-hosts-file\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036448 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-multus-certs\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.037846 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036468 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036488 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036493 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-multus-certs\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036498 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-kubernetes\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036526 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036555 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-registration-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036552 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-script-lib\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036559 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-kubelet\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036579 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08d074a5-dadb-4e71-a67f-9bc5c33d8857-iptables-alerter-script\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.036623 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036624 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-registration-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.036669 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs podName:33574457-8ee3-4f93-80b7-3e961da45892 nodeName:}" failed. No retries permitted until 2026-11-02 06:15:53.536653768 +0000 UTC m=+2.291465776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs") pod "network-metrics-daemon-5wc8x" (UID: "33574457-8ee3-4f93-80b7-3e961da45892") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036680 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70e61026-bb17-490c-aed0-c7387729c9ef-serviceca\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-device-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.038680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036762 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036811 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-multus\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.035865 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1edfb6b-cbff-47e6-aab2-a942731747b8-tmp-dir\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-agent-certs\") pod \"konnectivity-agent-fc9xt\" (UID: \"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad\") " pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036874 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-konnectivity-ca\") pod \"konnectivity-agent-fc9xt\" (UID: \"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad\") " pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036890 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036913 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-etc-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036934 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovn-node-metrics-cert\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-dbus\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.036979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-node-log\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037001 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-config\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysconfig\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037050 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-conf\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-kubelet\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-slash\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-env-overrides\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037160 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.039489 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037228 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-node-log\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037242 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08d074a5-dadb-4e71-a67f-9bc5c33d8857-iptables-alerter-script\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037573 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70e61026-bb17-490c-aed0-c7387729c9ef-serviceca\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037633 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037655 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-dbus\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037704 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysconfig\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037747 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-slash\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037764 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-conf\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037786 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-kubelet\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-kubelet-config\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037823 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-netd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037874 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037895 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-systemd-units\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037914 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-systemd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037935 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zc6b\" (UniqueName: \"kubernetes.io/projected/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-kube-api-access-9zc6b\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037956 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dkw\" (UniqueName: \"kubernetes.io/projected/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kube-api-access-24dkw\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.040064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.037978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-netns\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038023 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038046 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-bin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038051 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-config\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-conf-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038158 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-conf-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038313 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-netns\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038360 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-bin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038366 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-netd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038401 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-systemd-units\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038466 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038496 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-kubelet-config\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038509 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-systemd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038542 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038551 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-etc-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.038704 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-env-overrides\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.040548 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.039174 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.041004 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.039436 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.041004 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.039958 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-tmp\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.041004 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.039976 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovn-node-metrics-cert\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.041004 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.039985 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-etc-tuned\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.041004 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.040097 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-agent-certs\") pod \"konnectivity-agent-fc9xt\" (UID: \"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad\") " pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:15:53.088567 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.088301 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:15:53.088567 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.088332 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:15:53.088567 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.088348 2574 projected.go:196] Error preparing data for projected volume kube-api-access-zljkr for pod openshift-network-diagnostics/network-check-target-k7c7h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:53.088567 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.088417 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr podName:ec8433d1-0c6c-4732-b8a4-d73c251cd5ef nodeName:}" failed. No retries permitted until 2026-11-02 06:15:53.588396507 +0000 UTC m=+2.343208517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zljkr" (UniqueName: "kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr") pod "network-check-target-k7c7h" (UID: "ec8433d1-0c6c-4732-b8a4-d73c251cd5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:53.089985 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.089952 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwvv\" (UniqueName: \"kubernetes.io/projected/70e61026-bb17-490c-aed0-c7387729c9ef-kube-api-access-jkwvv\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:15:53.090315 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.090290 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw9sf\" (UniqueName: \"kubernetes.io/projected/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-kube-api-access-sw9sf\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.090725 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.090706 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw58c\" (UniqueName: \"kubernetes.io/projected/e1edfb6b-cbff-47e6-aab2-a942731747b8-kube-api-access-dw58c\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:15:53.091024 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.091005 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tbp\" (UniqueName: \"kubernetes.io/projected/513b68de-5cb3-4038-9592-5eb8989af256-kube-api-access-96tbp\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.091435 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.091411 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghc4z\" (UniqueName: \"kubernetes.io/projected/08d074a5-dadb-4e71-a67f-9bc5c33d8857-kube-api-access-ghc4z\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:15:53.091435 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.091426 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p667p\" (UniqueName: \"kubernetes.io/projected/33574457-8ee3-4f93-80b7-3e961da45892-kube-api-access-p667p\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:53.091559 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.091545 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zc6b\" (UniqueName: \"kubernetes.io/projected/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-kube-api-access-9zc6b\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.091751 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.091734 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dkw\" (UniqueName: \"kubernetes.io/projected/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kube-api-access-24dkw\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.092151 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.092131 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tsjq\" (UniqueName: \"kubernetes.io/projected/aed3a242-64df-41a6-98b3-e2a6543918fb-kube-api-access-6tsjq\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.181147 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.181118 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:15:53.202763 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.202743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:15:53.220316 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.220296 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c9z9s" Nov 02 06:15:53.237849 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.237828 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:15:53.266226 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.266207 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:15:53.268360 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.268345 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:15:53.284880 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.284858 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:15:53.320297 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.320279 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:15:53.322446 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.322421 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:15:53.503639 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:53.503600 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c68cd55c462db386dbcd1795bbdd95.slice/crio-54fac0e94f1ae66415d08d23dcdb220f55f5c91d40d835f010524156d691d681 WatchSource:0}: Error finding container 54fac0e94f1ae66415d08d23dcdb220f55f5c91d40d835f010524156d691d681: Status 404 returned error can't find the container with id 54fac0e94f1ae66415d08d23dcdb220f55f5c91d40d835f010524156d691d681 Nov 02 06:15:53.504126 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:53.504103 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceea1ff964927afa3d04bca3afe81300.slice/crio-dd00b934f62eaf7987253945786fd750109f151bc6ddc3af9308ab1a0b45915d WatchSource:0}: Error finding container dd00b934f62eaf7987253945786fd750109f151bc6ddc3af9308ab1a0b45915d: Status 404 returned error can't find the container with id dd00b934f62eaf7987253945786fd750109f151bc6ddc3af9308ab1a0b45915d Nov 02 06:15:53.508873 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.508850 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:15:53.543071 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.543042 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:53.543228 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.543110 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:53.543228 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.543201 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:53.543228 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.543204 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:53.543348 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.543249 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret podName:6ab6077a-6df9-4de9-8632-98830b9a8653 nodeName:}" failed. No retries permitted until 2026-11-02 06:15:54.543234384 +0000 UTC m=+3.298046389 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret") pod "global-pull-secret-syncer-4vv5l" (UID: "6ab6077a-6df9-4de9-8632-98830b9a8653") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:53.543348 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.543264 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs podName:33574457-8ee3-4f93-80b7-3e961da45892 nodeName:}" failed. No retries permitted until 2026-11-02 06:15:54.543256947 +0000 UTC m=+3.298068937 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs") pod "network-metrics-daemon-5wc8x" (UID: "33574457-8ee3-4f93-80b7-3e961da45892") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:53.643992 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.643964 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljkr\" (UniqueName: \"kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr\") pod \"network-check-target-k7c7h\" (UID: \"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef\") " pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:15:53.644448 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.644132 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:15:53.644448 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.644148 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:15:53.644448 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.644156 2574 projected.go:196] Error preparing data for projected volume kube-api-access-zljkr for pod openshift-network-diagnostics/network-check-target-k7c7h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:53.644448 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:53.644203 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr podName:ec8433d1-0c6c-4732-b8a4-d73c251cd5ef nodeName:}" failed. No retries permitted until 2026-11-02 06:15:54.644189338 +0000 UTC m=+3.399001333 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zljkr" (UniqueName: "kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr") pod "network-check-target-k7c7h" (UID: "ec8433d1-0c6c-4732-b8a4-d73c251cd5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:53.764870 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.764772 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" event={"ID":"ceea1ff964927afa3d04bca3afe81300","Type":"ContainerStarted","Data":"dd00b934f62eaf7987253945786fd750109f151bc6ddc3af9308ab1a0b45915d"} Nov 02 06:15:53.765945 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.765921 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" event={"ID":"b2c68cd55c462db386dbcd1795bbdd95","Type":"ContainerStarted","Data":"54fac0e94f1ae66415d08d23dcdb220f55f5c91d40d835f010524156d691d681"} Nov 02 06:15:53.888225 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.888196 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:53.965755 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.965726 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:53.995925 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.995863 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 06:10:52 +0000 UTC" deadline="2028-08-04 13:25:15.213795656 +0000 UTC" Nov 02 06:15:53.995925 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:53.995916 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15391h9m21.217882702s" Nov 02 06:15:54.069599 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:54.069559 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1edfb6b_cbff_47e6_aab2_a942731747b8.slice/crio-103a381d095d3dc8460db7dc3b9f7f1bb32c7eff5a3ae623958ad578843603b7 WatchSource:0}: Error finding container 103a381d095d3dc8460db7dc3b9f7f1bb32c7eff5a3ae623958ad578843603b7: Status 404 returned error can't find the container with id 103a381d095d3dc8460db7dc3b9f7f1bb32c7eff5a3ae623958ad578843603b7 Nov 02 06:15:54.094871 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:54.094842 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf129d6cd_acad_478c_a2a7_cfa37d4c39f9.slice/crio-cce6f0dcf89ed25853c811b5a966b3dbb03a7cb0e97fe2f57dce90002c39a066 WatchSource:0}: Error finding container cce6f0dcf89ed25853c811b5a966b3dbb03a7cb0e97fe2f57dce90002c39a066: Status 404 returned error can't find the container with id cce6f0dcf89ed25853c811b5a966b3dbb03a7cb0e97fe2f57dce90002c39a066 Nov 02 06:15:54.114309 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.114287 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:15:54.126714 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:54.126690 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11477d6f_1bd9_40c9_8d57_c0a0e1edaebf.slice/crio-515cdd6d108cd258a8c18f932eee2aab666b253613d210693c976da0e42721c4 WatchSource:0}: Error finding container 515cdd6d108cd258a8c18f932eee2aab666b253613d210693c976da0e42721c4: Status 404 returned error can't find the container with id 515cdd6d108cd258a8c18f932eee2aab666b253613d210693c976da0e42721c4 Nov 02 06:15:54.143044 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:54.143020 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513b68de_5cb3_4038_9592_5eb8989af256.slice/crio-4583c5bf2f09110b4e9f31ec1ea5fe1aac02a6a603fda289b5f80240f7e4a11f WatchSource:0}: Error finding container 4583c5bf2f09110b4e9f31ec1ea5fe1aac02a6a603fda289b5f80240f7e4a11f: Status 404 returned error can't find the container with id 4583c5bf2f09110b4e9f31ec1ea5fe1aac02a6a603fda289b5f80240f7e4a11f Nov 02 06:15:54.159356 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:54.159334 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d074a5_dadb_4e71_a67f_9bc5c33d8857.slice/crio-afe8c4db14574370b926606db8d90243b5bf4189d3df575039d31638e0272024 WatchSource:0}: Error finding container afe8c4db14574370b926606db8d90243b5bf4189d3df575039d31638e0272024: Status 404 returned error can't find the container with id afe8c4db14574370b926606db8d90243b5bf4189d3df575039d31638e0272024 Nov 02 06:15:54.176601 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:54.176574 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e61026_bb17_490c_aed0_c7387729c9ef.slice/crio-01f0dc5e329df98637af2a43e2be37461f376413021346e2aa38421b0324b096 WatchSource:0}: Error finding container 01f0dc5e329df98637af2a43e2be37461f376413021346e2aa38421b0324b096: Status 404 returned error can't find the container with id 01f0dc5e329df98637af2a43e2be37461f376413021346e2aa38421b0324b096 Nov 02 06:15:54.222768 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:54.222735 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c85b19_43b7_4bb1_8bb3_0e1ff3006fad.slice/crio-984c9dd494487b444a38d6b0a72ef146fc1e402598de8b4f32e185c2f3b906ca WatchSource:0}: Error finding container 984c9dd494487b444a38d6b0a72ef146fc1e402598de8b4f32e185c2f3b906ca: Status 404 returned error can't find the container with id 984c9dd494487b444a38d6b0a72ef146fc1e402598de8b4f32e185c2f3b906ca Nov 02 06:15:54.223298 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:54.223277 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3100b5d8_88b0_45c8_8e06_d758f2b769fa.slice/crio-d3ce2ea3476675e0f61cf5abcdb5c2eb572cc98c5a6ee422cbf9afb08427dc41 WatchSource:0}: Error finding container d3ce2ea3476675e0f61cf5abcdb5c2eb572cc98c5a6ee422cbf9afb08427dc41: Status 404 returned error can't find the container with id d3ce2ea3476675e0f61cf5abcdb5c2eb572cc98c5a6ee422cbf9afb08427dc41 Nov 02 06:15:54.254433 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:15:54.254411 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed3a242_64df_41a6_98b3_e2a6543918fb.slice/crio-53d88e33402ed0776711a72509ab90f16abc4f74cad8462acb0570335a99b621 WatchSource:0}: Error finding container 53d88e33402ed0776711a72509ab90f16abc4f74cad8462acb0570335a99b621: Status 404 returned error can't find the container with id 53d88e33402ed0776711a72509ab90f16abc4f74cad8462acb0570335a99b621 Nov 02 06:15:54.550154 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.550126 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:54.550315 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.550179 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:54.550315 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:54.550261 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:54.550315 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:54.550264 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:54.550315 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:54.550305 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret podName:6ab6077a-6df9-4de9-8632-98830b9a8653 nodeName:}" failed. No retries permitted until 2026-11-02 06:15:56.550294052 +0000 UTC m=+5.305106057 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret") pod "global-pull-secret-syncer-4vv5l" (UID: "6ab6077a-6df9-4de9-8632-98830b9a8653") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:54.550450 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:54.550319 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs podName:33574457-8ee3-4f93-80b7-3e961da45892 nodeName:}" failed. No retries permitted until 2026-11-02 06:15:56.550312436 +0000 UTC m=+5.305124426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs") pod "network-metrics-daemon-5wc8x" (UID: "33574457-8ee3-4f93-80b7-3e961da45892") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:54.650710 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.650682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljkr\" (UniqueName: \"kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr\") pod \"network-check-target-k7c7h\" (UID: \"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef\") " pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:15:54.651143 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:54.650865 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:15:54.651143 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:54.650887 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:15:54.651143 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:54.650900 2574 projected.go:196] Error preparing data for projected volume kube-api-access-zljkr for pod openshift-network-diagnostics/network-check-target-k7c7h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:54.651143 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:54.650962 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr podName:ec8433d1-0c6c-4732-b8a4-d73c251cd5ef nodeName:}" failed. No retries permitted until 2026-11-02 06:15:56.650942582 +0000 UTC m=+5.405754587 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zljkr" (UniqueName: "kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr") pod "network-check-target-k7c7h" (UID: "ec8433d1-0c6c-4732-b8a4-d73c251cd5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:54.761441 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.761404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:15:54.761603 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.761404 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:54.761603 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:54.761508 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:15:54.761724 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.761415 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:54.761724 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:54.761614 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:15:54.761819 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:54.761725 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:15:54.769050 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.769023 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fc9xt" event={"ID":"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad","Type":"ContainerStarted","Data":"984c9dd494487b444a38d6b0a72ef146fc1e402598de8b4f32e185c2f3b906ca"} Nov 02 06:15:54.770425 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.770401 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-58ckp" event={"ID":"70e61026-bb17-490c-aed0-c7387729c9ef","Type":"ContainerStarted","Data":"01f0dc5e329df98637af2a43e2be37461f376413021346e2aa38421b0324b096"} Nov 02 06:15:54.772213 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.772187 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8tczj" event={"ID":"08d074a5-dadb-4e71-a67f-9bc5c33d8857","Type":"ContainerStarted","Data":"afe8c4db14574370b926606db8d90243b5bf4189d3df575039d31638e0272024"} Nov 02 06:15:54.773633 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.773609 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" event={"ID":"513b68de-5cb3-4038-9592-5eb8989af256","Type":"ContainerStarted","Data":"4583c5bf2f09110b4e9f31ec1ea5fe1aac02a6a603fda289b5f80240f7e4a11f"} Nov 02 06:15:54.775064 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.775043 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c9z9s" event={"ID":"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf","Type":"ContainerStarted","Data":"515cdd6d108cd258a8c18f932eee2aab666b253613d210693c976da0e42721c4"} Nov 02 06:15:54.776444 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.776421 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" event={"ID":"f129d6cd-acad-478c-a2a7-cfa37d4c39f9","Type":"ContainerStarted","Data":"cce6f0dcf89ed25853c811b5a966b3dbb03a7cb0e97fe2f57dce90002c39a066"} Nov 02 06:15:54.777726 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.777699 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5jr5c" event={"ID":"e1edfb6b-cbff-47e6-aab2-a942731747b8","Type":"ContainerStarted","Data":"103a381d095d3dc8460db7dc3b9f7f1bb32c7eff5a3ae623958ad578843603b7"} Nov 02 06:15:54.779454 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.779426 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" event={"ID":"aed3a242-64df-41a6-98b3-e2a6543918fb","Type":"ContainerStarted","Data":"53d88e33402ed0776711a72509ab90f16abc4f74cad8462acb0570335a99b621"} Nov 02 06:15:54.780858 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.780835 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" event={"ID":"3100b5d8-88b0-45c8-8e06-d758f2b769fa","Type":"ContainerStarted","Data":"d3ce2ea3476675e0f61cf5abcdb5c2eb572cc98c5a6ee422cbf9afb08427dc41"} Nov 02 06:15:54.996974 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.996912 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 06:10:52 +0000 UTC" deadline="2028-07-12 02:34:53.235025158 +0000 UTC" Nov 02 06:15:54.996974 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:54.996974 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14828h18m58.238056987s" Nov 02 06:15:56.564899 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:56.564860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:56.565532 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:56.564917 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:56.565532 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:56.565069 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:56.565532 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:56.565146 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs podName:33574457-8ee3-4f93-80b7-3e961da45892 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:00.565127665 +0000 UTC m=+9.319939670 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs") pod "network-metrics-daemon-5wc8x" (UID: "33574457-8ee3-4f93-80b7-3e961da45892") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:15:56.565532 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:56.565218 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:56.565532 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:56.565251 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret podName:6ab6077a-6df9-4de9-8632-98830b9a8653 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:00.565241043 +0000 UTC m=+9.320053053 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret") pod "global-pull-secret-syncer-4vv5l" (UID: "6ab6077a-6df9-4de9-8632-98830b9a8653") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:15:56.665887 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:56.665841 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljkr\" (UniqueName: \"kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr\") pod \"network-check-target-k7c7h\" (UID: \"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef\") " pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:15:56.666149 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:56.666130 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:15:56.666239 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:56.666160 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:15:56.666239 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:56.666173 2574 projected.go:196] Error preparing data for projected volume kube-api-access-zljkr for pod openshift-network-diagnostics/network-check-target-k7c7h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:56.666239 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:56.666235 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr podName:ec8433d1-0c6c-4732-b8a4-d73c251cd5ef nodeName:}" failed. No retries permitted until 2026-11-02 06:16:00.666217899 +0000 UTC m=+9.421029914 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zljkr" (UniqueName: "kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr") pod "network-check-target-k7c7h" (UID: "ec8433d1-0c6c-4732-b8a4-d73c251cd5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:15:56.760961 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:56.760887 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:56.761146 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:56.761031 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:15:56.761146 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:56.761111 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:56.761260 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:56.761229 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:15:56.761325 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:56.761296 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:15:56.761438 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:56.761416 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:15:58.762107 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:58.760936 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:15:58.762107 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:58.761289 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:15:58.762107 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:58.761519 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:15:58.762107 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:58.761617 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:15:58.762107 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:58.761687 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:15:58.762107 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:15:58.761756 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:15:58.806763 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:58.806718 2574 generic.go:356] "Generic (PLEG): container finished" podID="b2c68cd55c462db386dbcd1795bbdd95" containerID="a33b745b35d223959b65ba8bfaa6ac4bd7c23b2effed05921ec421101747eb2b" exitCode=0 Nov 02 06:15:58.806928 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:58.806809 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" event={"ID":"b2c68cd55c462db386dbcd1795bbdd95","Type":"ContainerDied","Data":"a33b745b35d223959b65ba8bfaa6ac4bd7c23b2effed05921ec421101747eb2b"} Nov 02 06:15:58.814285 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:58.814255 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" event={"ID":"ceea1ff964927afa3d04bca3afe81300","Type":"ContainerStarted","Data":"00fb99159cbcde00a716abba6907c68f546a94ece424f79c22dc72ff0520c19f"} Nov 02 06:15:58.817321 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:58.817292 2574 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="977eb3b23839066d993507f6eadbc2884b4e168ea1de4644f4c01e1fb1e95457" exitCode=0 Nov 02 06:15:58.817433 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:58.817336 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" event={"ID":"aed3a242-64df-41a6-98b3-e2a6543918fb","Type":"ContainerDied","Data":"977eb3b23839066d993507f6eadbc2884b4e168ea1de4644f4c01e1fb1e95457"} Nov 02 06:15:59.121899 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:59.119691 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" podStartSLOduration=7.119669523 podStartE2EDuration="7.119669523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:59.014520719 +0000 UTC m=+7.769332724" watchObservedRunningTime="2026-11-02 06:15:59.119669523 +0000 UTC m=+7.874481540" Nov 02 06:15:59.830862 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:59.830151 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fc9xt" event={"ID":"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad","Type":"ContainerStarted","Data":"337dd42d97e156ae5d82a53ed6ed2b2939523797b7e6cd723e4da18cb783736c"} Nov 02 06:15:59.835337 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:15:59.835247 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" event={"ID":"b2c68cd55c462db386dbcd1795bbdd95","Type":"ContainerStarted","Data":"ce46fb649f248e52d089caa3308babb18f287083a951347dfb1fdb2b7d94dccb"} Nov 02 06:16:00.039752 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:00.039690 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fc9xt" podStartSLOduration=4.411430389 podStartE2EDuration="9.039668857s" totalImagesPullingTime="4.628238468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:15:59.93538405 +0000 UTC m=+8.690196063" watchObservedRunningTime="2026-11-02 06:16:00.039668857 +0000 UTC m=+8.794480873" Nov 02 06:16:00.597978 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:00.597931 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:00.598182 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:00.597995 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:00.598253 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:00.598237 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:00.598460 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:00.598442 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret podName:6ab6077a-6df9-4de9-8632-98830b9a8653 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:08.598329597 +0000 UTC m=+17.353141612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret") pod "global-pull-secret-syncer-4vv5l" (UID: "6ab6077a-6df9-4de9-8632-98830b9a8653") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:00.598665 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:00.598527 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:00.598665 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:00.598576 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs podName:33574457-8ee3-4f93-80b7-3e961da45892 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:08.598559474 +0000 UTC m=+17.353371479 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs") pod "network-metrics-daemon-5wc8x" (UID: "33574457-8ee3-4f93-80b7-3e961da45892") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:00.699410 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:00.699036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljkr\" (UniqueName: \"kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr\") pod \"network-check-target-k7c7h\" (UID: \"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef\") " pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:00.699410 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:00.699289 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:16:00.699410 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:00.699322 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:16:00.699410 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:00.699344 2574 projected.go:196] Error preparing data for projected volume kube-api-access-zljkr for pod openshift-network-diagnostics/network-check-target-k7c7h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:00.704823 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:00.700356 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr podName:ec8433d1-0c6c-4732-b8a4-d73c251cd5ef nodeName:}" failed. No retries permitted until 2026-11-02 06:16:08.700143107 +0000 UTC m=+17.454955113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zljkr" (UniqueName: "kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr") pod "network-check-target-k7c7h" (UID: "ec8433d1-0c6c-4732-b8a4-d73c251cd5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:00.761702 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:00.761264 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:00.761702 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:00.761408 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:00.761702 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:00.761483 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:00.761702 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:00.761551 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:00.761702 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:00.761599 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:00.761702 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:00.761661 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:00.839433 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:00.839363 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-58ckp" event={"ID":"70e61026-bb17-490c-aed0-c7387729c9ef","Type":"ContainerStarted","Data":"3a338397d1c78d55a8dc7efea05e5e390515c5b766ca350790b1859b41c65b9f"} Nov 02 06:16:00.942683 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:00.942627 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" podStartSLOduration=8.942606482 podStartE2EDuration="8.942606482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:00.04046971 +0000 UTC m=+8.795281724" watchObservedRunningTime="2026-11-02 06:16:00.942606482 +0000 UTC m=+9.697418496" Nov 02 06:16:01.845830 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:01.845792 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" event={"ID":"3100b5d8-88b0-45c8-8e06-d758f2b769fa","Type":"ContainerStarted","Data":"b8ff8c0e79992a6ea33414eac3b50f7a3542f27149046b1780459a1de3ad5ed9"} Nov 02 06:16:02.761999 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:02.761913 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:02.762197 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:02.762061 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:02.762197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:02.762134 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:02.762197 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:02.762186 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:02.762315 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:02.762239 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:02.762346 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:02.762315 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:02.850462 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:02.850432 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c9z9s" event={"ID":"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf","Type":"ContainerStarted","Data":"220dbce97eb97fa6e764a2a5fff6ba288b0f1bbee7985f1d898c77d27253b6f4"} Nov 02 06:16:02.852055 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:02.852032 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5jr5c" event={"ID":"e1edfb6b-cbff-47e6-aab2-a942731747b8","Type":"ContainerStarted","Data":"f603257712a6ef1717f2c964cb6009da904b3eca8ea891ced19b48fd01d8ba69"} Nov 02 06:16:02.930954 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:02.930934 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 06:16:02.955098 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:02.955028 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-c9z9s" podStartSLOduration=3.851685495 podStartE2EDuration="11.955011189s" totalImagesPullingTime="8.103325694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:02.954962808 +0000 UTC m=+11.709774846" watchObservedRunningTime="2026-11-02 06:16:02.955011189 +0000 UTC m=+11.709823196" Nov 02 06:16:02.955858 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:02.955826 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-58ckp" podStartSLOduration=6.118610656 podStartE2EDuration="11.955815069s" totalImagesPullingTime="5.837204413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:00.94357059 +0000 UTC m=+9.698382604" watchObservedRunningTime="2026-11-02 06:16:02.955815069 +0000 UTC m=+11.710627082" Nov 02 06:16:03.057901 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:03.057790 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5jr5c" podStartSLOduration=3.7699078679999998 podStartE2EDuration="12.057770784s" totalImagesPullingTime="8.287862916s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:03.05750125 +0000 UTC m=+11.812313264" watchObservedRunningTime="2026-11-02 06:16:03.057770784 +0000 UTC m=+11.812582799" Nov 02 06:16:03.697252 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:03.697132 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T06:16:02.931046686Z","UUID":"f98790f6-c0e5-443f-ac60-e2e6f3d1ce86","Handler":null,"Name":"","Endpoint":""} Nov 02 06:16:03.699181 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:03.699157 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 06:16:03.699307 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:03.699187 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 06:16:03.856906 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:03.856797 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" event={"ID":"3100b5d8-88b0-45c8-8e06-d758f2b769fa","Type":"ContainerStarted","Data":"816362973bdd189ca416e4d42118b999eda627e277eaf204b7eacf0b390309f3"} Nov 02 06:16:04.436681 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:04.436642 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:16:04.437402 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:04.437352 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:16:04.761366 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:04.761123 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:04.761509 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:04.761482 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:04.761584 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:04.761161 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:04.761630 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:04.761594 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:04.761630 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:04.761151 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:04.761726 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:04.761667 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:04.860651 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:04.860619 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" event={"ID":"aed3a242-64df-41a6-98b3-e2a6543918fb","Type":"ContainerStarted","Data":"3eddefc7360763a88ff20dc569bb110ad190a0cdc37bed7ea1e0b1e1a0cc1b5b"} Nov 02 06:16:04.862318 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:04.862285 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8tczj" event={"ID":"08d074a5-dadb-4e71-a67f-9bc5c33d8857","Type":"ContainerStarted","Data":"23eb10179e3cf441fc28db07d9412ca15462e1c8767acfb8fb00ae8ac6ba110d"} Nov 02 06:16:04.862466 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:04.862452 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:16:04.862943 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:04.862928 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:16:05.083833 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:05.083775 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8tczj" podStartSLOduration=5.668426532 podStartE2EDuration="14.083758441s" totalImagesPullingTime="8.415331909s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:05.082974728 +0000 UTC m=+13.837786741" watchObservedRunningTime="2026-11-02 06:16:05.083758441 +0000 UTC m=+13.838570456" Nov 02 06:16:05.866370 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:05.866334 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" event={"ID":"513b68de-5cb3-4038-9592-5eb8989af256","Type":"ContainerStarted","Data":"0b6110f3cb2df69b98cda7b5b5b58f87118931746e289c984676198d09eb5ef6"} Nov 02 06:16:05.868184 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:05.868163 2574 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="3eddefc7360763a88ff20dc569bb110ad190a0cdc37bed7ea1e0b1e1a0cc1b5b" exitCode=0 Nov 02 06:16:05.868270 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:05.868232 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" event={"ID":"aed3a242-64df-41a6-98b3-e2a6543918fb","Type":"ContainerDied","Data":"3eddefc7360763a88ff20dc569bb110ad190a0cdc37bed7ea1e0b1e1a0cc1b5b"} Nov 02 06:16:05.870243 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:05.870223 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" event={"ID":"3100b5d8-88b0-45c8-8e06-d758f2b769fa","Type":"ContainerStarted","Data":"a76aca1771cacd94003f69e44ce67e1917daa904a7ca81b5c588e66596c920b8"} Nov 02 06:16:05.977640 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:05.977576 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" podStartSLOduration=3.853778932 podStartE2EDuration="14.977561967s" totalImagesPullingTime="11.123783035s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:05.977135159 +0000 UTC m=+14.731947171" watchObservedRunningTime="2026-11-02 06:16:05.977561967 +0000 UTC m=+14.732373979" Nov 02 06:16:06.198389 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:06.198337 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" podStartSLOduration=4.54277835 podStartE2EDuration="15.198321913s" totalImagesPullingTime="10.655543563s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:06.08802065 +0000 UTC m=+14.842832662" watchObservedRunningTime="2026-11-02 06:16:06.198321913 +0000 UTC m=+14.953133926" Nov 02 06:16:06.760722 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:06.760696 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:06.760722 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:06.760721 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:06.760969 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:06.760704 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:06.760969 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:06.760808 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:06.760969 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:06.760891 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:06.760969 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:06.760941 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:07.877869 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:07.877612 2574 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="d40959ac1049e22ac8752cbd05c9cf457ddb54f16b8886bb5dd366e348c7580a" exitCode=0 Nov 02 06:16:07.878448 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:07.877671 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" event={"ID":"aed3a242-64df-41a6-98b3-e2a6543918fb","Type":"ContainerDied","Data":"d40959ac1049e22ac8752cbd05c9cf457ddb54f16b8886bb5dd366e348c7580a"} Nov 02 06:16:08.663508 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:08.663471 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:08.663689 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:08.663537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:08.663689 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:08.663597 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:08.663689 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:08.663667 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:08.663689 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:08.663678 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret podName:6ab6077a-6df9-4de9-8632-98830b9a8653 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:24.663658498 +0000 UTC m=+33.418470494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret") pod "global-pull-secret-syncer-4vv5l" (UID: "6ab6077a-6df9-4de9-8632-98830b9a8653") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:08.663847 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:08.663713 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs podName:33574457-8ee3-4f93-80b7-3e961da45892 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:24.663701532 +0000 UTC m=+33.418513524 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs") pod "network-metrics-daemon-5wc8x" (UID: "33574457-8ee3-4f93-80b7-3e961da45892") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:08.761095 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:08.761047 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:08.761259 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:08.761056 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:08.761259 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:08.761206 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:08.761259 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:08.761068 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:08.761429 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:08.761230 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:08.761429 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:08.761352 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:08.764197 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:08.764176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljkr\" (UniqueName: \"kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr\") pod \"network-check-target-k7c7h\" (UID: \"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef\") " pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:08.764349 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:08.764333 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 02 06:16:08.764390 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:08.764355 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 02 06:16:08.764390 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:08.764369 2574 projected.go:196] Error preparing data for projected volume kube-api-access-zljkr for pod openshift-network-diagnostics/network-check-target-k7c7h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:08.764455 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:08.764419 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr podName:ec8433d1-0c6c-4732-b8a4-d73c251cd5ef nodeName:}" failed. No retries permitted until 2026-11-02 06:16:24.764403398 +0000 UTC m=+33.519215391 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zljkr" (UniqueName: "kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr") pod "network-check-target-k7c7h" (UID: "ec8433d1-0c6c-4732-b8a4-d73c251cd5ef") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 02 06:16:09.885635 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:09.885548 2574 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="e9e19b2cfc554d7e00e5dbae24e934bd03c91fbeec43c90c96e4aea47dc5639c" exitCode=0 Nov 02 06:16:09.885635 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:09.885613 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" event={"ID":"aed3a242-64df-41a6-98b3-e2a6543918fb","Type":"ContainerDied","Data":"e9e19b2cfc554d7e00e5dbae24e934bd03c91fbeec43c90c96e4aea47dc5639c"} Nov 02 06:16:10.760602 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:10.760566 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:10.760602 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:10.760603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:10.760856 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:10.760605 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:10.760856 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:10.760687 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:10.760856 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:10.760813 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:10.760973 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:10.760903 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:11.893090 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:11.892897 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" event={"ID":"f129d6cd-acad-478c-a2a7-cfa37d4c39f9","Type":"ContainerStarted","Data":"31a5faf7ab8a7a9a8b2260267c9a8a0c19be4d8188f0ee9f6810d6ebc7fec85a"} Nov 02 06:16:11.893422 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:11.893152 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" event={"ID":"f129d6cd-acad-478c-a2a7-cfa37d4c39f9","Type":"ContainerStarted","Data":"d9e0ba47df6201267322cf502184ff4d00f688cd2381fc7a3253ce2a1315fa79"} Nov 02 06:16:11.893422 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:11.893205 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" event={"ID":"f129d6cd-acad-478c-a2a7-cfa37d4c39f9","Type":"ContainerStarted","Data":"ecc6a29a54a0f96bbeacbae9841bf048ce68a9a7c794718dde9840770017216b"} Nov 02 06:16:11.893422 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:11.893255 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" event={"ID":"f129d6cd-acad-478c-a2a7-cfa37d4c39f9","Type":"ContainerStarted","Data":"583394e67d824f0cf88a9a005d5997639c35f1b8785c57b57d799a37aba46104"} Nov 02 06:16:11.893422 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:11.893298 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" event={"ID":"f129d6cd-acad-478c-a2a7-cfa37d4c39f9","Type":"ContainerStarted","Data":"f01e0829108aa97904a6412d63ac48334ba83c405bb439eafc45ccf8399a5337"} Nov 02 06:16:12.307987 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:12.307962 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:12.579233 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:12.579158 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:12.760415 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:12.760386 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:12.760601 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:12.760468 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:12.760601 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:12.760486 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:12.760601 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:12.760583 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:12.760717 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:12.760701 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:12.760771 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:12.760755 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:12.897358 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:12.897278 2574 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="1b911b79378594ea3f5a1d7f97b1d425574912ad1157d71b7030e792e8f7add1" exitCode=0 Nov 02 06:16:12.897919 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:12.897354 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" event={"ID":"aed3a242-64df-41a6-98b3-e2a6543918fb","Type":"ContainerDied","Data":"1b911b79378594ea3f5a1d7f97b1d425574912ad1157d71b7030e792e8f7add1"} Nov 02 06:16:12.900139 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:12.900114 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" event={"ID":"f129d6cd-acad-478c-a2a7-cfa37d4c39f9","Type":"ContainerStarted","Data":"20e815a732f8a8ef4f997b75c95dc4a0489f38b7f1e8edb4b08d408e5829e149"} Nov 02 06:16:13.008664 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:13.008637 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:13.230090 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:13.230057 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:13.419513 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:13.419482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:13.615307 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:13.615231 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:13.904709 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:13.904634 2574 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="6a9da6ac149a0047caccb04cc43d21e54c7e2b6616286972590b06b293574459" exitCode=0 Nov 02 06:16:13.904709 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:13.904673 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" event={"ID":"aed3a242-64df-41a6-98b3-e2a6543918fb","Type":"ContainerDied","Data":"6a9da6ac149a0047caccb04cc43d21e54c7e2b6616286972590b06b293574459"} Nov 02 06:16:14.761295 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:14.761267 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:14.761295 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:14.761294 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:14.761578 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:14.761366 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:14.761578 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:14.761414 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:14.761578 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:14.761473 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:14.761578 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:14.761561 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:14.910437 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:14.910405 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" event={"ID":"aed3a242-64df-41a6-98b3-e2a6543918fb","Type":"ContainerStarted","Data":"eebef3d931c85886a8db282acb2dd59fa7f7f6475924c59e387ee000ac43e04d"} Nov 02 06:16:14.913501 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:14.913476 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" event={"ID":"f129d6cd-acad-478c-a2a7-cfa37d4c39f9","Type":"ContainerStarted","Data":"1de5c00a57bbce776d7e8c1e9991ea2618eab91958b1d70daf75946c8a469a3f"} Nov 02 06:16:15.066180 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:15.066067 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gm9nl" podStartSLOduration=9.72211828 podStartE2EDuration="24.066054632s" totalImagesPullingTime="14.343936352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:15.065599622 +0000 UTC m=+23.820411635" watchObservedRunningTime="2026-11-02 06:16:15.066054632 +0000 UTC m=+23.820866645" Nov 02 06:16:15.229123 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:15.229091 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:15.403448 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:15.403359 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:16.760806 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:16.760778 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:16.760806 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:16.760806 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:16.761398 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:16.760813 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:16.761398 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:16.760878 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:16.761398 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:16.760938 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:16.761398 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:16.761004 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:16.920617 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:16.920583 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" event={"ID":"f129d6cd-acad-478c-a2a7-cfa37d4c39f9","Type":"ContainerStarted","Data":"0f0714cfe55261831e7e383b741d6a4fba8938185665d6ce71947c2346735d63"} Nov 02 06:16:16.920858 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:16.920836 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:16:16.920927 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:16.920900 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:16:16.936110 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:16.936090 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:16:17.110128 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:17.110036 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:17.170843 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:17.170785 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" podStartSLOduration=8.707474001 podStartE2EDuration="26.170773196s" totalImagesPullingTime="17.463299195s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:17.154100213 +0000 UTC m=+25.908912225" watchObservedRunningTime="2026-11-02 06:16:17.170773196 +0000 UTC m=+25.925585208" Nov 02 06:16:17.289324 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:17.289289 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:17.922982 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:17.922801 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:16:17.936871 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:17.936847 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:16:18.761176 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:18.761093 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:18.761361 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:18.761199 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:18.761361 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:18.761102 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:18.761361 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:18.761239 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:18.761361 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:18.761314 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:18.761519 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:18.761388 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:18.779636 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:18.779614 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k7c7h"] Nov 02 06:16:18.782589 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:18.782564 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4vv5l"] Nov 02 06:16:18.783201 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:18.783184 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5wc8x"] Nov 02 06:16:18.924131 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:18.924099 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:18.924559 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:18.924338 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:18.924559 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:18.924423 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:18.924559 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:18.924454 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:18.924559 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:18.924522 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:18.924749 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:18.924656 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:19.933330 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:19.933302 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:20.116848 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:20.116818 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:20.760957 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:20.760927 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:20.761115 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:20.761001 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:20.761115 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:20.761025 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:20.761216 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:20.761161 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:20.761285 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:20.761256 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:20.761360 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:20.761340 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:21.118659 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:21.118585 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:21.290524 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:21.290494 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:21.469579 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:21.469549 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:21.711034 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:21.711003 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:21.912726 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:21.912703 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:22.597443 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:22.597412 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:22.760574 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:22.760545 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:22.760717 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:22.760584 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:22.760717 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:22.760545 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:22.760717 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:22.760655 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wc8x" podUID="33574457-8ee3-4f93-80b7-3e961da45892" Nov 02 06:16:22.760812 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:22.760755 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4vv5l" podUID="6ab6077a-6df9-4de9-8632-98830b9a8653" Nov 02 06:16:22.760845 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:22.760821 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k7c7h" podUID="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" Nov 02 06:16:22.799161 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:22.799130 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:22.966089 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:22.966044 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5jr5c_e1edfb6b-cbff-47e6-aab2-a942731747b8/dns-node-resolver/0.log" Nov 02 06:16:22.997030 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:22.997002 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:23.199329 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:23.199298 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:23.385345 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:23.385266 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:23.580616 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:23.580587 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:23.764599 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:23.764570 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-58ckp_70e61026-bb17-490c-aed0-c7387729c9ef/node-ca/0.log" Nov 02 06:16:24.059912 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.059841 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeReady" Nov 02 06:16:24.060050 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.059966 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 02 06:16:24.176115 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.176087 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:24.559502 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.559470 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:24.675913 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.675882 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:24.675913 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.675918 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:24.676112 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:24.676022 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:24.676112 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:24.676024 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:24.676112 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:24.676072 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs podName:33574457-8ee3-4f93-80b7-3e961da45892 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:56.676058629 +0000 UTC m=+65.430870619 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs") pod "network-metrics-daemon-5wc8x" (UID: "33574457-8ee3-4f93-80b7-3e961da45892") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 02 06:16:24.676112 ip-10-0-130-34 kubenswrapper[2574]: E1102 06:16:24.676100 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret podName:6ab6077a-6df9-4de9-8632-98830b9a8653 nodeName:}" failed. No retries permitted until 2026-11-02 06:16:56.676093954 +0000 UTC m=+65.430905945 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret") pod "global-pull-secret-syncer-4vv5l" (UID: "6ab6077a-6df9-4de9-8632-98830b9a8653") : object "kube-system"/"original-pull-secret" not registered Nov 02 06:16:24.749933 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.749904 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:24.760887 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.760868 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:24.760982 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.760890 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:24.760982 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.760869 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:24.776685 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.776665 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljkr\" (UniqueName: \"kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr\") pod \"network-check-target-k7c7h\" (UID: \"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef\") " pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:24.809920 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.809854 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 06:16:24.809920 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.809859 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 06:16:24.820589 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.820566 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljkr\" (UniqueName: \"kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr\") pod \"network-check-target-k7c7h\" (UID: \"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef\") " pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:24.825690 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.825670 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 06:16:24.856009 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.855988 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bfjz\"" Nov 02 06:16:24.856009 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.856001 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 06:16:24.856148 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.855997 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-cdk7k\"" Nov 02 06:16:24.856148 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.856056 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2646z\"" Nov 02 06:16:24.860145 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.860124 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:24.946220 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:24.946186 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:25.159380 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:25.159295 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:25.284695 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:25.284659 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k7c7h"] Nov 02 06:16:25.289964 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:16:25.289938 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8433d1_0c6c_4732_b8a4_d73c251cd5ef.slice/crio-720138a2f707f417b09f73428a540e0cb23544375175df98084dd21057007244 WatchSource:0}: Error finding container 720138a2f707f417b09f73428a540e0cb23544375175df98084dd21057007244: Status 404 returned error can't find the container with id 720138a2f707f417b09f73428a540e0cb23544375175df98084dd21057007244 Nov 02 06:16:25.367593 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:25.367561 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:25.762644 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:25.762609 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:25.939058 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:25.938997 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k7c7h" event={"ID":"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef","Type":"ContainerStarted","Data":"720138a2f707f417b09f73428a540e0cb23544375175df98084dd21057007244"} Nov 02 06:16:25.942418 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:25.942395 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:26.319742 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:26.319707 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:26.497394 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:26.497360 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:26.693196 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:26.693165 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:26.864855 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:26.864822 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:27.113961 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:27.113886 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:27.917710 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:27.917675 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:28.324701 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:28.324632 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:28.509547 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:28.509518 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:28.712021 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:28.711992 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:28.910579 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:28.910536 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:30.029322 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:30.029282 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:30.202483 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:30.202446 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:30.377851 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:30.377725 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:30.555490 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:30.555456 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:30.727668 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:30.727634 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:30.917783 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:30.917754 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:31.952715 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:31.952676 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k7c7h" event={"ID":"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef","Type":"ContainerStarted","Data":"2ab64d27b097c0c95044980c833a7f26fe7d07ca6a71988f371018e668cbb475"} Nov 02 06:16:31.953112 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:31.952817 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:16:32.057794 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:32.057741 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k7c7h" podStartSLOduration=35.058372619 podStartE2EDuration="41.057725807s" totalImagesPullingTime="5.999353188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:32.056546576 +0000 UTC m=+40.811358589" watchObservedRunningTime="2026-11-02 06:16:32.057725807 +0000 UTC m=+40.812537810" Nov 02 06:16:32.121747 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:32.121711 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:32.318547 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:32.318452 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:32.521969 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:32.521913 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:32.718274 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:32.718247 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:32.924997 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:32.924961 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:34.117464 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:34.117427 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:34.325504 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:34.325471 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:34.516219 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:34.516184 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:34.720511 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:34.720454 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:34.916903 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:34.916871 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:36.684796 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:36.684765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:36.863460 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:36.863430 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:41.074365 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.074337 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc"] Nov 02 06:16:41.077273 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.077257 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-x9xwq"] Nov 02 06:16:41.077398 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.077380 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc" Nov 02 06:16:41.079509 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.079488 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-x48kz"] Nov 02 06:16:41.079626 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.079602 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:16:41.081294 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.081274 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.120499 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.120464 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b4b56fff9-77j92"] Nov 02 06:16:41.122167 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.122147 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wvjp6"] Nov 02 06:16:41.122280 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.122262 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.123947 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.123928 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.125987 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.125969 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 06:16:41.126086 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.125995 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 06:16:41.126370 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.126356 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kqjnz\"" Nov 02 06:16:41.127588 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.127573 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 06:16:41.127727 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.127713 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 06:16:41.137043 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.137026 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 06:16:41.137163 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.137066 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 06:16:41.137163 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.137108 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 06:16:41.137236 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.137178 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 06:16:41.143641 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.143620 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:41.166608 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.166576 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc"] Nov 02 06:16:41.170992 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.170970 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9xwq"] Nov 02 06:16:41.175897 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.175845 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 06:16:41.186985 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.186964 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-data-volume\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.187065 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.186993 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.187065 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.187020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.187156 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.187144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-crio-socket\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.187196 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.187169 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc4fn\" (UniqueName: \"kubernetes.io/projected/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-api-access-dc4fn\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.187196 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.187188 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-cert\") pod \"ingress-canary-x9xwq\" (UID: \"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e\") " pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:16:41.187261 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.187205 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxbtw\" (UniqueName: \"kubernetes.io/projected/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-kube-api-access-hxbtw\") pod \"ingress-canary-x9xwq\" (UID: \"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e\") " pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:16:41.187300 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.187261 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nxgs\" (UniqueName: \"kubernetes.io/projected/d1842ed7-0d43-4f3d-815b-a2f45026a569-kube-api-access-4nxgs\") pod \"migrator-66df5d496b-qx2rc\" (UID: \"d1842ed7-0d43-4f3d-815b-a2f45026a569\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc" Nov 02 06:16:41.222770 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.222737 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 06:16:41.223936 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.223917 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x48kz"] Nov 02 06:16:41.224481 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.224463 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b4b56fff9-77j92"] Nov 02 06:16:41.225034 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.225017 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wvjp6"] Nov 02 06:16:41.229173 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.229148 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 06:16:41.229265 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.229207 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 06:16:41.229265 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.229208 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 06:16:41.229265 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.229217 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxd68\"" Nov 02 06:16:41.229360 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.229279 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 06:16:41.229360 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.229224 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rkkc6\"" Nov 02 06:16:41.229360 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.229288 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 06:16:41.229442 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.229359 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rpvjg\"" Nov 02 06:16:41.229703 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.229682 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75dck\"" Nov 02 06:16:41.288107 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.288286 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288128 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-crio-socket\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.288286 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4fn\" (UniqueName: \"kubernetes.io/projected/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-api-access-dc4fn\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.288286 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288165 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lq9q\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-kube-api-access-7lq9q\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.288286 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288185 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-cert\") pod \"ingress-canary-x9xwq\" (UID: \"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e\") " pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:16:41.288286 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288202 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-image-registry-private-configuration\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.288286 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288218 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-registry-tls\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.288286 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288237 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbtw\" (UniqueName: \"kubernetes.io/projected/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-kube-api-access-hxbtw\") pod \"ingress-canary-x9xwq\" (UID: \"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e\") " pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:16:41.288609 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nxgs\" (UniqueName: \"kubernetes.io/projected/d1842ed7-0d43-4f3d-815b-a2f45026a569-kube-api-access-4nxgs\") pod \"migrator-66df5d496b-qx2rc\" (UID: \"d1842ed7-0d43-4f3d-815b-a2f45026a569\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc" Nov 02 06:16:41.288609 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288449 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-trusted-ca\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.288609 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc512a04-0325-41ec-b022-010c44a7c96d-config-volume\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.288609 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288510 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc512a04-0325-41ec-b022-010c44a7c96d-tmp-dir\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.288609 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288524 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-crio-socket\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.288609 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-bound-sa-token\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.288876 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288624 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc512a04-0325-41ec-b022-010c44a7c96d-metrics-tls\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.288876 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288663 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmcw6\" (UniqueName: \"kubernetes.io/projected/fc512a04-0325-41ec-b022-010c44a7c96d-kube-api-access-qmcw6\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.288876 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288744 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.288876 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288747 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-data-volume\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.288876 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-installation-pull-secrets\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.288876 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288821 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.288876 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288840 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-registry-certificates\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.289168 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288882 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/563834d8-a7ee-490b-a123-7358961378a2-ca-trust-extracted\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.289168 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.288992 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-data-volume\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.290726 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.290701 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-cert\") pod \"ingress-canary-x9xwq\" (UID: \"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e\") " pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:16:41.290856 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.290839 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.341910 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.341843 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4fn\" (UniqueName: \"kubernetes.io/projected/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-api-access-dc4fn\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.342017 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.341965 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nxgs\" (UniqueName: \"kubernetes.io/projected/d1842ed7-0d43-4f3d-815b-a2f45026a569-kube-api-access-4nxgs\") pod \"migrator-66df5d496b-qx2rc\" (UID: \"d1842ed7-0d43-4f3d-815b-a2f45026a569\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc" Nov 02 06:16:41.342388 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.342366 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbtw\" (UniqueName: \"kubernetes.io/projected/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-kube-api-access-hxbtw\") pod \"ingress-canary-x9xwq\" (UID: \"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e\") " pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:16:41.344660 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.344638 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:41.387992 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.387948 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc" Nov 02 06:16:41.389213 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389190 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lq9q\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-kube-api-access-7lq9q\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.389317 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-image-registry-private-configuration\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.389317 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-registry-tls\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.389317 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389288 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-trusted-ca\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.389317 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389308 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc512a04-0325-41ec-b022-010c44a7c96d-config-volume\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.389504 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389428 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc512a04-0325-41ec-b022-010c44a7c96d-tmp-dir\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.389504 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389469 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-bound-sa-token\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.389590 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389547 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc512a04-0325-41ec-b022-010c44a7c96d-metrics-tls\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.389590 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389582 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcw6\" (UniqueName: \"kubernetes.io/projected/fc512a04-0325-41ec-b022-010c44a7c96d-kube-api-access-qmcw6\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.389685 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389626 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-installation-pull-secrets\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.389685 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389662 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-registry-certificates\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.389785 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389690 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/563834d8-a7ee-490b-a123-7358961378a2-ca-trust-extracted\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.389785 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.389758 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc512a04-0325-41ec-b022-010c44a7c96d-tmp-dir\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.390287 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.390112 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/563834d8-a7ee-490b-a123-7358961378a2-ca-trust-extracted\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.390287 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.390228 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc512a04-0325-41ec-b022-010c44a7c96d-config-volume\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.390537 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.390515 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-trusted-ca\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.390794 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.390759 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-registry-certificates\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.392314 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.392289 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-installation-pull-secrets\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.392409 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.392289 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-image-registry-private-configuration\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.392409 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.392301 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc512a04-0325-41ec-b022-010c44a7c96d-metrics-tls\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.392512 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.392481 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-registry-tls\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.394228 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.394213 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:16:41.400595 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.400549 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:16:41.445177 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.445113 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lq9q\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-kube-api-access-7lq9q\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.447451 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.447224 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-bound-sa-token\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.450901 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.449656 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcw6\" (UniqueName: \"kubernetes.io/projected/fc512a04-0325-41ec-b022-010c44a7c96d-kube-api-access-qmcw6\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.553986 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.553952 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:41.732271 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.732245 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:41.737896 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.737879 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:41.743643 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.743614 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:41.764841 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.764817 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-x9xwq"] Nov 02 06:16:41.767567 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:16:41.767540 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15fd4393_b2d4_4f12_8ae8_f38b92a3b05e.slice/crio-bcdf3f05b5f67f1c900a64f48063888b423afaf680d80124cb8a06e144af1045 WatchSource:0}: Error finding container bcdf3f05b5f67f1c900a64f48063888b423afaf680d80124cb8a06e144af1045: Status 404 returned error can't find the container with id bcdf3f05b5f67f1c900a64f48063888b423afaf680d80124cb8a06e144af1045 Nov 02 06:16:41.805621 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.805518 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc"] Nov 02 06:16:41.810219 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:16:41.810186 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1842ed7_0d43_4f3d_815b_a2f45026a569.slice/crio-79ddf8fc3d2f1674daf4034a5fc144fed4eeb7bbf05689cf8499c8f258043156 WatchSource:0}: Error finding container 79ddf8fc3d2f1674daf4034a5fc144fed4eeb7bbf05689cf8499c8f258043156: Status 404 returned error can't find the container with id 79ddf8fc3d2f1674daf4034a5fc144fed4eeb7bbf05689cf8499c8f258043156 Nov 02 06:16:41.811687 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.811662 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-x48kz"] Nov 02 06:16:41.815447 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:16:41.815424 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba1e4bef_666b_4088_8e2b_9f7a55ab1327.slice/crio-234bf4a0078dbd451430f0a64a0e69814b4b8f68022a728ca2a97ae5b6b8b8a3 WatchSource:0}: Error finding container 234bf4a0078dbd451430f0a64a0e69814b4b8f68022a728ca2a97ae5b6b8b8a3: Status 404 returned error can't find the container with id 234bf4a0078dbd451430f0a64a0e69814b4b8f68022a728ca2a97ae5b6b8b8a3 Nov 02 06:16:41.945948 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.945913 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:41.977680 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.977650 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9xwq" event={"ID":"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e","Type":"ContainerStarted","Data":"bcdf3f05b5f67f1c900a64f48063888b423afaf680d80124cb8a06e144af1045"} Nov 02 06:16:41.978703 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.978684 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc" event={"ID":"d1842ed7-0d43-4f3d-815b-a2f45026a569","Type":"ContainerStarted","Data":"79ddf8fc3d2f1674daf4034a5fc144fed4eeb7bbf05689cf8499c8f258043156"} Nov 02 06:16:41.980019 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.979995 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x48kz" event={"ID":"ba1e4bef-666b-4088-8e2b-9f7a55ab1327","Type":"ContainerStarted","Data":"21b5cca9134771d0614a891c59d6ebfb8b5bc854fad79a22b80aa9f46d4d118d"} Nov 02 06:16:41.980127 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:41.980041 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x48kz" event={"ID":"ba1e4bef-666b-4088-8e2b-9f7a55ab1327","Type":"ContainerStarted","Data":"234bf4a0078dbd451430f0a64a0e69814b4b8f68022a728ca2a97ae5b6b8b8a3"} Nov 02 06:16:42.097512 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:42.097402 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b4b56fff9-77j92"] Nov 02 06:16:42.100455 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:42.100426 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wvjp6"] Nov 02 06:16:42.100562 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:16:42.100455 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod563834d8_a7ee_490b_a123_7358961378a2.slice/crio-455abc622b558391826c4d80530a6bfdac8370de346c4ad12a140b2a3df8e060 WatchSource:0}: Error finding container 455abc622b558391826c4d80530a6bfdac8370de346c4ad12a140b2a3df8e060: Status 404 returned error can't find the container with id 455abc622b558391826c4d80530a6bfdac8370de346c4ad12a140b2a3df8e060 Nov 02 06:16:42.103420 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:16:42.103401 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc512a04_0325_41ec_b022_010c44a7c96d.slice/crio-472aaa6688c7b077955b4d9402776a09e43da3e06484d27f42bf129a82fe550a WatchSource:0}: Error finding container 472aaa6688c7b077955b4d9402776a09e43da3e06484d27f42bf129a82fe550a: Status 404 returned error can't find the container with id 472aaa6688c7b077955b4d9402776a09e43da3e06484d27f42bf129a82fe550a Nov 02 06:16:42.985503 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:42.985145 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" event={"ID":"563834d8-a7ee-490b-a123-7358961378a2","Type":"ContainerStarted","Data":"411c75c479e84c15e4d05762db436662ed987bf69e6c7e49feaceb58d1e52403"} Nov 02 06:16:42.985503 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:42.985420 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" event={"ID":"563834d8-a7ee-490b-a123-7358961378a2","Type":"ContainerStarted","Data":"455abc622b558391826c4d80530a6bfdac8370de346c4ad12a140b2a3df8e060"} Nov 02 06:16:42.985503 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:42.985465 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:16:42.990613 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:42.990574 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wvjp6" event={"ID":"fc512a04-0325-41ec-b022-010c44a7c96d","Type":"ContainerStarted","Data":"472aaa6688c7b077955b4d9402776a09e43da3e06484d27f42bf129a82fe550a"} Nov 02 06:16:43.995217 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:43.995134 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x48kz" event={"ID":"ba1e4bef-666b-4088-8e2b-9f7a55ab1327","Type":"ContainerStarted","Data":"3c994b6145eaa31f4514d9e8b769687d08a1c84f751751cd9031bf61530752e3"} Nov 02 06:16:43.996809 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:43.996781 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc" event={"ID":"d1842ed7-0d43-4f3d-815b-a2f45026a569","Type":"ContainerStarted","Data":"12a252d5e79480d646426968b62eb0ed5a1c416d49f25ac4e2e067d24348f2c3"} Nov 02 06:16:44.103219 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:44.103133 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc" podStartSLOduration=1.230802596 podStartE2EDuration="3.10311358s" totalImagesPullingTime="1.872310984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:44.102348451 +0000 UTC m=+52.857160486" watchObservedRunningTime="2026-11-02 06:16:44.10311358 +0000 UTC m=+52.857925594" Nov 02 06:16:44.103417 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:44.103363 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" podStartSLOduration=3.103354857 podStartE2EDuration="3.103354857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:43.099051545 +0000 UTC m=+51.853863560" watchObservedRunningTime="2026-11-02 06:16:44.103354857 +0000 UTC m=+52.858166870" Nov 02 06:16:44.410101 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:44.410013 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:44.596768 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:44.596723 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:46.004089 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:46.004035 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-x48kz" event={"ID":"ba1e4bef-666b-4088-8e2b-9f7a55ab1327","Type":"ContainerStarted","Data":"8eb52cfaa07b1572e6b97c152e6c138582dd6f2df01211263b58afe0df74eb5c"} Nov 02 06:16:46.005843 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:46.005822 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wvjp6" event={"ID":"fc512a04-0325-41ec-b022-010c44a7c96d","Type":"ContainerStarted","Data":"42a2425a4e7f535edb9e1a90aed0a4a2a28ce4dd8e3d2dcc4af1948b60deed0f"} Nov 02 06:16:46.005918 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:46.005869 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wvjp6" event={"ID":"fc512a04-0325-41ec-b022-010c44a7c96d","Type":"ContainerStarted","Data":"3361034f5b1262b6832b271fdb8cf2494f339bd592dbef4526fdeea8ecf6ca76"} Nov 02 06:16:46.005993 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:46.005980 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:46.157760 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:46.157680 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-x48kz" podStartSLOduration=1.6302656679999998 podStartE2EDuration="5.157662407s" totalImagesPullingTime="3.527396739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 06:16:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:46.11159805 +0000 UTC m=+54.866410064" watchObservedRunningTime="2026-11-02 06:16:46.157662407 +0000 UTC m=+54.912474423" Nov 02 06:16:46.219129 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:46.219056 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-wvjp6" podStartSLOduration=2.38853905 podStartE2EDuration="5.219042354s" totalImagesPullingTime="2.830503304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:46.218656941 +0000 UTC m=+54.973468954" watchObservedRunningTime="2026-11-02 06:16:46.219042354 +0000 UTC m=+54.973854367" Nov 02 06:16:47.316809 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:47.316771 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:47.513374 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:47.513345 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:48.012952 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:48.012915 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-x9xwq" event={"ID":"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e","Type":"ContainerStarted","Data":"90527400bd22456dda28f68fc9489aab018b845643c74548f326789c5a3f6f82"} Nov 02 06:16:48.118472 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:48.118418 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-x9xwq" podStartSLOduration=1.5051796419999999 podStartE2EDuration="7.118404566s" totalImagesPullingTime="5.613224924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:16:48.117954674 +0000 UTC m=+56.872766689" watchObservedRunningTime="2026-11-02 06:16:48.118404566 +0000 UTC m=+56.873216593" Nov 02 06:16:49.118346 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:49.118313 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:49.305510 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:49.305480 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:49.485303 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:49.485273 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:49.936483 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:49.936460 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:16:50.190164 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:50.190057 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:50.386480 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:50.386443 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:50.566576 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:50.566490 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:50.746257 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:50.746226 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:51.689213 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:51.689184 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:51.870272 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:51.870243 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:52.161990 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:52.161916 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:52.344301 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:52.344271 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:52.531850 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:52.531803 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:52.722466 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:52.722417 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:52.946798 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:52.946769 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:53.140917 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:53.140882 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:53.320921 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:53.320839 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:55.062719 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:55.062683 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:55.237191 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:55.237148 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:55.475015 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:55.474986 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:55.654239 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:55.654202 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:55.835921 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:55.835826 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:56.010384 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:56.010359 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wvjp6" Nov 02 06:16:56.014179 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:56.014152 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:56.704682 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:56.704649 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:56.705044 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:56.704711 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:56.798068 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:56.798031 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 06:16:56.798068 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:56.798059 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 06:16:56.807412 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:56.807385 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:56.807535 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:56.807385 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:56.969050 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:56.968974 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2646z\"" Nov 02 06:16:56.972020 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:56.971999 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:16:57.005358 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:57.005334 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bfjz\"" Nov 02 06:16:57.008487 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:57.008462 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:16:57.326907 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:57.326822 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5wc8x"] Nov 02 06:16:57.329265 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:16:57.329226 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33574457_8ee3_4f93_80b7_3e961da45892.slice/crio-76a3f856faa0da3530937b29b7b962169bf14e976e7da03e6420b93b99761364 WatchSource:0}: Error finding container 76a3f856faa0da3530937b29b7b962169bf14e976e7da03e6420b93b99761364: Status 404 returned error can't find the container with id 76a3f856faa0da3530937b29b7b962169bf14e976e7da03e6420b93b99761364 Nov 02 06:16:57.372972 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:57.372948 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4vv5l"] Nov 02 06:16:57.375237 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:16:57.375215 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab6077a_6df9_4de9_8632_98830b9a8653.slice/crio-489ed7a9cca070d71d9dfabec5a2bcd1191e2efd5ccf43ae1e9349bb1bd763ec WatchSource:0}: Error finding container 489ed7a9cca070d71d9dfabec5a2bcd1191e2efd5ccf43ae1e9349bb1bd763ec: Status 404 returned error can't find the container with id 489ed7a9cca070d71d9dfabec5a2bcd1191e2efd5ccf43ae1e9349bb1bd763ec Nov 02 06:16:57.827943 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:57.827913 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:58.006710 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:58.006680 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:58.041722 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:58.041688 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4vv5l" event={"ID":"6ab6077a-6df9-4de9-8632-98830b9a8653","Type":"ContainerStarted","Data":"489ed7a9cca070d71d9dfabec5a2bcd1191e2efd5ccf43ae1e9349bb1bd763ec"} Nov 02 06:16:58.042979 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:58.042953 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5wc8x" event={"ID":"33574457-8ee3-4f93-80b7-3e961da45892","Type":"ContainerStarted","Data":"76a3f856faa0da3530937b29b7b962169bf14e976e7da03e6420b93b99761364"} Nov 02 06:16:59.227299 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:59.227269 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:16:59.418268 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:16:59.418235 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:00.051126 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:00.051092 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5wc8x" event={"ID":"33574457-8ee3-4f93-80b7-3e961da45892","Type":"ContainerStarted","Data":"afbbdbb4305fe44a65fd33516539401b046bd6882d9815651724d8fd69ec7073"} Nov 02 06:17:00.051287 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:00.051147 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5wc8x" event={"ID":"33574457-8ee3-4f93-80b7-3e961da45892","Type":"ContainerStarted","Data":"c63b7b70e916a2290aede9cf1945c24b6fc23081403b282f25e1679a45f31e01"} Nov 02 06:17:00.157394 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:00.157342 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5wc8x" podStartSLOduration=67.472459145 podStartE2EDuration="1m9.157326597s" totalImagesPullingTime="1.684867452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:00.155852156 +0000 UTC m=+68.910664173" watchObservedRunningTime="2026-11-02 06:17:00.157326597 +0000 UTC m=+68.912138610" Nov 02 06:17:00.277419 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:00.277387 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:00.454820 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:00.454679 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:01.646436 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:01.646394 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:01.838521 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:01.838483 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:02.029624 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.029590 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:02.242759 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.242729 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:02.395590 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.395509 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p7227"] Nov 02 06:17:02.401051 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.401029 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.427631 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.427601 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:02.441334 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.441304 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.441456 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.441345 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/720c25e0-1117-4a48-af60-9022daed412a-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.441456 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.441377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97d22\" (UniqueName: \"kubernetes.io/projected/720c25e0-1117-4a48-af60-9022daed412a-kube-api-access-97d22\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.441563 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.441471 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.498456 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.498424 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 06:17:02.498456 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.498446 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cglw4\"" Nov 02 06:17:02.498456 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.498459 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 06:17:02.498706 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.498515 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 06:17:02.498706 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.498525 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 06:17:02.498706 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.498590 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 06:17:02.499552 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.499521 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p7227"] Nov 02 06:17:02.542093 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.542040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.542093 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.542089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/720c25e0-1117-4a48-af60-9022daed412a-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.542319 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.542109 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97d22\" (UniqueName: \"kubernetes.io/projected/720c25e0-1117-4a48-af60-9022daed412a-kube-api-access-97d22\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.542319 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.542131 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.542950 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.542922 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/720c25e0-1117-4a48-af60-9022daed412a-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.544759 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.544737 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.545608 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.545586 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7wr46"] Nov 02 06:17:02.545726 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.545702 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.562949 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.562921 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.598162 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.598135 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97d22\" (UniqueName: \"kubernetes.io/projected/720c25e0-1117-4a48-af60-9022daed412a-kube-api-access-97d22\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.613942 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.613915 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:02.643182 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.643149 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-wtmp\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.643311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.643197 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.643311 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.643293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-textfile\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.643413 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.643351 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.643413 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.643380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-metrics-client-ca\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.643413 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.643410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-root\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.643505 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.643485 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-sys\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.643554 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.643513 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-tls\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.643554 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.643534 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42bxk\" (UniqueName: \"kubernetes.io/projected/0122975d-2172-4e2a-9af8-47f972bc5ecc-kube-api-access-42bxk\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.649646 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.649597 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 06:17:02.665964 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.665937 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xbcc6\"" Nov 02 06:17:02.666113 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.666087 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 06:17:02.666261 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.666236 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 06:17:02.720790 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.720732 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:02.744175 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.744092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-wtmp\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.744175 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.744145 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.744325 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.744176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-textfile\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.744325 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.744210 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.744325 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.744229 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-metrics-client-ca\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.744325 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.744259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-root\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.745428 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.745404 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-sys\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.745527 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.745447 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-tls\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.745527 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.745475 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42bxk\" (UniqueName: \"kubernetes.io/projected/0122975d-2172-4e2a-9af8-47f972bc5ecc-kube-api-access-42bxk\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.745619 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.745541 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-wtmp\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.745673 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.745662 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-textfile\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.745743 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.745718 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-sys\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.745859 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.745838 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-root\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.746068 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.746049 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.746174 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.746152 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-metrics-client-ca\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.748234 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.748186 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-tls\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.749618 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.749594 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.801775 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.801742 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42bxk\" (UniqueName: \"kubernetes.io/projected/0122975d-2172-4e2a-9af8-47f972bc5ecc-kube-api-access-42bxk\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.822348 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.822316 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:02.921808 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.921785 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:02.950883 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:17:02.950849 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0122975d_2172_4e2a_9af8_47f972bc5ecc.slice/crio-2954891028fe653bf4c8bfb6184c864a609ee50c0e464ab179d93495605de5dc WatchSource:0}: Error finding container 2954891028fe653bf4c8bfb6184c864a609ee50c0e464ab179d93495605de5dc: Status 404 returned error can't find the container with id 2954891028fe653bf4c8bfb6184c864a609ee50c0e464ab179d93495605de5dc Nov 02 06:17:02.957930 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:02.957910 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:17:03.066644 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:03.066604 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wr46" event={"ID":"0122975d-2172-4e2a-9af8-47f972bc5ecc","Type":"ContainerStarted","Data":"2954891028fe653bf4c8bfb6184c864a609ee50c0e464ab179d93495605de5dc"} Nov 02 06:17:03.143022 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:03.142991 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-p7227"] Nov 02 06:17:03.146920 ip-10-0-130-34 kubenswrapper[2574]: W1102 06:17:03.146890 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720c25e0_1117_4a48_af60_9022daed412a.slice/crio-de292ec76e3378e0fe776106c92c847f2c77c16c288bf83db72f157011d745e8 WatchSource:0}: Error finding container de292ec76e3378e0fe776106c92c847f2c77c16c288bf83db72f157011d745e8: Status 404 returned error can't find the container with id de292ec76e3378e0fe776106c92c847f2c77c16c288bf83db72f157011d745e8 Nov 02 06:17:03.174094 ip-10-0-130-34 kubenswrapper[2574]: I1102 06:17:03.174011 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:03.483174 ip-10-0-130-34 systemd[1]: Stopping Kubernetes Kubelet... Nov 02 06:17:03.505426 ip-10-0-130-34 systemd[1]: kubelet.service: Deactivated successfully. Nov 02 06:17:03.505596 ip-10-0-130-34 systemd[1]: Stopped Kubernetes Kubelet. Nov 02 06:17:03.505646 ip-10-0-130-34 systemd[1]: kubelet.service: Consumed 3.457s CPU time. Nov 02 06:17:03.521735 ip-10-0-130-34 systemd[1]: Starting Kubernetes Kubelet... Nov 02 06:17:03.585287 ip-10-0-130-34 kubenswrapper[5025]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:17:03.585287 ip-10-0-130-34 kubenswrapper[5025]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 02 06:17:03.585287 ip-10-0-130-34 kubenswrapper[5025]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:17:03.585287 ip-10-0-130-34 kubenswrapper[5025]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 02 06:17:03.588196 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588108 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:03.588196 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588193 5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:03.588196 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588197 5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:03.588196 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588200 5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:03.588196 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588203 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588206 5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588209 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588212 5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588214 5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588217 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588220 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588223 5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588226 5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588229 5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588232 5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588234 5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588237 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588239 5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588242 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588244 5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588247 5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588249 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588252 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588255 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:03.588385 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588258 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588260 5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588263 5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588266 5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588268 5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588271 5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588274 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588284 5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588288 5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588291 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588293 5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588296 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588299 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588302 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588304 5025 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588307 5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588310 5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588314 5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588319 5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:03.588907 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588323 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588326 5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588329 5025 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588332 5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588335 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588338 5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588340 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588345 5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588349 5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588352 5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588354 5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588357 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588360 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588362 5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588365 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588368 5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588371 5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588373 5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588376 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:03.589491 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588379 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588381 5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588384 5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588387 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588390 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588392 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588395 5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588398 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588401 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588404 5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588406 5025 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588409 5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588412 5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588415 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588418 5025 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588421 5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588423 5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588426 5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588429 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588431 5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:03.589978 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588434 5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588438 5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588441 5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588443 5025 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588446 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588449 5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588451 5025 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588454 5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588457 5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588459 5025 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588462 5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588465 5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588467 5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588470 5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588472 5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588476 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588478 5025 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588481 5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588484 5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588487 5025 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588489 5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:03.590502 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588492 5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588498 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.588501 5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589276 5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589287 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589290 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589293 5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589296 5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589299 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589301 5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589304 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589307 5025 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589310 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589314 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589317 5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589319 5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589328 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589331 5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589334 5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589336 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:03.591107 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589339 5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589342 5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589344 5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589347 5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589350 5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589353 5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589356 5025 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589358 5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589361 5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589364 5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589366 5025 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589369 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589372 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589375 5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589377 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589380 5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589384 5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589386 5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589389 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:03.591614 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589392 5025 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589394 5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589397 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589400 5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589402 5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589405 5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589408 5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589411 5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589413 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589416 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589419 5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589422 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589425 5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589428 5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589430 5025 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589433 5025 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589435 5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589438 5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589440 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589443 5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:03.592124 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589446 5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589449 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589452 5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589454 5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589457 5025 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589459 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589462 5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589465 5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589467 5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589471 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589474 5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589476 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589479 5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589482 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589484 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589487 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589489 5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589492 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589495 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589498 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:03.592745 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589501 5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589504 5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589506 5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589509 5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589511 5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589515 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589517 5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589520 5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589522 5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589525 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589529 5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589534 5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589539 5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589542 5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589545 5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589547 5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589550 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589552 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:03.593307 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589555 5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589557 5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589560 5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589564 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589567 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589569 5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589572 5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589574 5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589577 5025 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589580 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589582 5025 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.589586 5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590117 5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590125 5025 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590134 5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590137 5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590140 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590143 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590146 5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590149 5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:03.593767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590154 5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590158 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590162 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590164 5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590167 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590170 5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590173 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590176 5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590179 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590181 5025 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590184 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590187 5025 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590189 5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590192 5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590195 5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590199 5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590202 5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590205 5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590207 5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590210 5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:03.594372 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590213 5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590215 5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590218 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590220 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590223 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590225 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590228 5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590236 5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590240 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590242 5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590245 5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590248 5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590250 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590253 5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590255 5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590258 5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590261 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590263 5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590266 5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:03.594917 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590269 5025 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590271 5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590285 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590288 5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590291 5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590293 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590296 5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590299 5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590305 5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590307 5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590310 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590313 5025 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590315 5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590318 5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590321 5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590323 5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590326 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590328 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590331 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590333 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:03.595429 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590338 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590341 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590344 5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590347 5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590349 5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590352 5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590354 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590357 5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590359 5025 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590362 5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590364 5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590367 5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590370 5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590372 5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590375 5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590378 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590380 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590383 5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590386 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590388 5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:03.595932 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590392 5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590395 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590398 5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590400 5025 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590403 5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590406 5025 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590408 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590413 5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590416 5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590419 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590422 5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590425 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590428 5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590431 5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590434 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590437 5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590440 5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590442 5025 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.590445 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590535 5025 flags.go:64] FLAG: --address="0.0.0.0" Nov 02 06:17:03.596467 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590542 5025 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590548 5025 flags.go:64] FLAG: --anonymous-auth="true" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590564 5025 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590569 5025 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590572 5025 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590576 5025 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590581 5025 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590584 5025 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590587 5025 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590590 5025 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590594 5025 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590597 5025 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590601 5025 flags.go:64] FLAG: --cgroup-root="" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590605 5025 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590608 5025 flags.go:64] FLAG: --client-ca-file="" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590611 5025 flags.go:64] FLAG: --cloud-provider="external" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590614 5025 flags.go:64] FLAG: --cluster-dns="[]" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590619 5025 flags.go:64] FLAG: --cluster-domain="" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590622 5025 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590625 5025 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590628 5025 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590632 5025 flags.go:64] FLAG: --container-log-max-files="5" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590636 5025 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 02 06:17:03.596972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590639 5025 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590644 5025 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590647 5025 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590651 5025 flags.go:64] FLAG: --contention-profiling="false" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590655 5025 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590658 5025 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590661 5025 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590664 5025 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590668 5025 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590672 5025 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590675 5025 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590677 5025 flags.go:64] FLAG: --enable-load-reader="false" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590680 5025 flags.go:64] FLAG: --enable-server="true" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590683 5025 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590687 5025 flags.go:64] FLAG: --event-burst="100" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590691 5025 flags.go:64] FLAG: --event-qps="50" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590694 5025 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590697 5025 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590700 5025 flags.go:64] FLAG: --eviction-hard="" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590704 5025 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590707 5025 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590712 5025 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590715 5025 flags.go:64] FLAG: --eviction-soft="" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590718 5025 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590721 5025 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 02 06:17:03.597552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590724 5025 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590727 5025 flags.go:64] FLAG: --experimental-mounter-path="" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590730 5025 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590733 5025 flags.go:64] FLAG: --fail-swap-on="true" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590736 5025 flags.go:64] FLAG: --feature-gates="" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590740 5025 flags.go:64] FLAG: --file-check-frequency="20s" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590743 5025 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590746 5025 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590751 5025 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590754 5025 flags.go:64] FLAG: --healthz-port="10248" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590757 5025 flags.go:64] FLAG: --help="false" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590760 5025 flags.go:64] FLAG: --hostname-override="ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590764 5025 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590767 5025 flags.go:64] FLAG: --http-check-frequency="20s" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590770 5025 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590774 5025 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590777 5025 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590780 5025 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590783 5025 flags.go:64] FLAG: --image-service-endpoint="" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590786 5025 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590789 5025 flags.go:64] FLAG: --kube-api-burst="100" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590792 5025 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590795 5025 flags.go:64] FLAG: --kube-api-qps="50" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590798 5025 flags.go:64] FLAG: --kube-reserved="" Nov 02 06:17:03.598178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590801 5025 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590804 5025 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590807 5025 flags.go:64] FLAG: --kubelet-cgroups="" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590810 5025 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590814 5025 flags.go:64] FLAG: --lock-file="" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590817 5025 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590820 5025 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590823 5025 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590829 5025 flags.go:64] FLAG: --log-json-split-stream="false" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590832 5025 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590835 5025 flags.go:64] FLAG: --log-text-split-stream="false" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590837 5025 flags.go:64] FLAG: --logging-format="text" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590840 5025 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590844 5025 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590847 5025 flags.go:64] FLAG: --manifest-url="" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590849 5025 flags.go:64] FLAG: --manifest-url-header="" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590856 5025 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590859 5025 flags.go:64] FLAG: --max-open-files="1000000" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590863 5025 flags.go:64] FLAG: --max-pods="110" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590866 5025 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590869 5025 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590873 5025 flags.go:64] FLAG: --memory-manager-policy="None" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590876 5025 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590879 5025 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590882 5025 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 02 06:17:03.598883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590885 5025 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590892 5025 flags.go:64] FLAG: --node-status-max-images="50" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590895 5025 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590898 5025 flags.go:64] FLAG: --oom-score-adj="-999" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590901 5025 flags.go:64] FLAG: --pod-cidr="" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590904 5025 flags.go:64] FLAG: --pod-manifest-path="" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590907 5025 flags.go:64] FLAG: --pod-max-pids="-1" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590910 5025 flags.go:64] FLAG: --pods-per-core="0" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590913 5025 flags.go:64] FLAG: --port="10250" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590916 5025 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590919 5025 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0f3cf3049f5a29920" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590923 5025 flags.go:64] FLAG: --qos-reserved="" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590927 5025 flags.go:64] FLAG: --read-only-port="10255" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590930 5025 flags.go:64] FLAG: --register-node="true" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590933 5025 flags.go:64] FLAG: --register-with-taints="" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590937 5025 flags.go:64] FLAG: --registry-burst="10" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590940 5025 flags.go:64] FLAG: --registry-qps="5" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590943 5025 flags.go:64] FLAG: --reserved-cpus="" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590946 5025 flags.go:64] FLAG: --reserved-memory="" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590950 5025 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590953 5025 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590956 5025 flags.go:64] FLAG: --rotate-certificates="false" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590959 5025 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590963 5025 flags.go:64] FLAG: --runonce="false" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590966 5025 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590970 5025 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590973 5025 flags.go:64] FLAG: --seccomp-default="false" Nov 02 06:17:03.599513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590976 5025 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590979 5025 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590983 5025 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590986 5025 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590989 5025 flags.go:64] FLAG: --storage-driver-password="root" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590992 5025 flags.go:64] FLAG: --storage-driver-secure="false" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590995 5025 flags.go:64] FLAG: --storage-driver-table="stats" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.590998 5025 flags.go:64] FLAG: --storage-driver-user="root" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591001 5025 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591004 5025 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591007 5025 flags.go:64] FLAG: --system-cgroups="" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591010 5025 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591016 5025 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591018 5025 flags.go:64] FLAG: --tls-cert-file="" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591021 5025 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591025 5025 flags.go:64] FLAG: --tls-min-version="" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591029 5025 flags.go:64] FLAG: --tls-private-key-file="" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591032 5025 flags.go:64] FLAG: --topology-manager-policy="none" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591035 5025 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591038 5025 flags.go:64] FLAG: --topology-manager-scope="container" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591041 5025 flags.go:64] FLAG: --v="2" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591046 5025 flags.go:64] FLAG: --version="false" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591050 5025 flags.go:64] FLAG: --vmodule="" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591054 5025 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591057 5025 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 02 06:17:03.600506 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591221 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591226 5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591230 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591234 5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591237 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591240 5025 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591243 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591245 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591248 5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591251 5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591254 5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591257 5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591259 5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591262 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591265 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591267 5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591270 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591272 5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591275 5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591277 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:03.601463 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591280 5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591283 5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591287 5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591292 5025 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591295 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591298 5025 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591300 5025 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591303 5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591307 5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591311 5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591314 5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591317 5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591319 5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591322 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591325 5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591329 5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591332 5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591334 5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591337 5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:03.602269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591340 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591342 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591345 5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591348 5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591351 5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591354 5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591357 5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591359 5025 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591362 5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591365 5025 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591367 5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591370 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591373 5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591375 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591378 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591381 5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591384 5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591386 5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591389 5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591391 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:03.602820 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591394 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591396 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591399 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591401 5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591404 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591406 5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591409 5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591412 5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591415 5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591418 5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591420 5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591423 5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591426 5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591428 5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591431 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591433 5025 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591436 5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591439 5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591442 5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591444 5025 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:03.603684 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591447 5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591450 5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591452 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591455 5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591457 5025 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591460 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591462 5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591465 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591467 5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591470 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591473 5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591475 5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591478 5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591480 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591483 5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591486 5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591488 5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591491 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591494 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:03.604204 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591496 5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591500 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591503 5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591505 5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591508 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591510 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591513 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591515 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.591542 5025 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591578 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591582 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591585 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591588 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591591 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591594 5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591597 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:03.604813 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591599 5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591602 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591604 5025 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591607 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591610 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591612 5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591615 5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591618 5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591621 5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591623 5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591626 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591630 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591633 5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591636 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591638 5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591641 5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591644 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591647 5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591649 5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:03.605311 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591653 5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591658 5025 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591662 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591665 5025 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591668 5025 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591671 5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591675 5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591678 5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591681 5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591684 5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591687 5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591690 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591692 5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591695 5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591698 5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591701 5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591703 5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591706 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591708 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591711 5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:03.605946 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591714 5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591716 5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591719 5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591721 5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591725 5025 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591728 5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591732 5025 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591734 5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591737 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591740 5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591742 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591745 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591748 5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591750 5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591753 5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591756 5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591758 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591761 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591763 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591766 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:03.606457 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591768 5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591771 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591774 5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591776 5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591779 5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591781 5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591784 5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591787 5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591790 5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591792 5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591795 5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591797 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591800 5025 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591802 5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591805 5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591807 5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591811 5025 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591814 5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591817 5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591820 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:03.606942 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591823 5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591825 5025 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591828 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591831 5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591833 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591836 5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591839 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591842 5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591844 5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591847 5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591849 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591852 5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591855 5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591858 5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591861 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591864 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591867 5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591869 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:03.607527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591872 5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.591875 5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.595624 5025 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.595645 5025 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.595666 5025 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.595671 5025 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595725 5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595731 5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595734 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595738 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595741 5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595744 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595747 5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595750 5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595753 5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595756 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595758 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595761 5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595764 5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595766 5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:03.607958 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595769 5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595772 5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595775 5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595777 5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595780 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595783 5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595786 5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595789 5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595794 5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595797 5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595801 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595804 5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595808 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595811 5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595815 5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595818 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595821 5025 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595824 5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595826 5025 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:03.608474 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595829 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595832 5025 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595834 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595837 5025 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595840 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595842 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595845 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595847 5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595850 5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595852 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595855 5025 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595857 5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595860 5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595863 5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595865 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595868 5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595871 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595873 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595876 5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:03.608949 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595879 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595882 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595884 5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595887 5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595890 5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595892 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595896 5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595900 5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595904 5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595907 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595910 5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595913 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595916 5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595919 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595921 5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595924 5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595926 5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595929 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595931 5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:03.609425 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595934 5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595937 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595940 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595942 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595945 5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595948 5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595951 5025 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595953 5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595956 5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595959 5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595962 5025 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595964 5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595967 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595969 5025 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595972 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595974 5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595977 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595979 5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595982 5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:03.609888 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595985 5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595988 5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595991 5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595993 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595996 5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.595999 5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596002 5025 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596004 5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596007 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596009 5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596012 5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596014 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596017 5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596020 5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596022 5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596025 5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:03.610368 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.596052 5025 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596207 5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596212 5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596216 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596219 5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596222 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596225 5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596227 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596230 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596233 5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596235 5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596238 5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596240 5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596243 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596245 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596248 5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596250 5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596253 5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596256 5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:03.610785 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596258 5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596261 5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596263 5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596267 5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596269 5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596273 5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596277 5025 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596280 5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596283 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596285 5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596288 5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596291 5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596293 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596296 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596298 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596301 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596303 5025 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596306 5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596308 5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:03.611269 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596311 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596314 5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596317 5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596319 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596322 5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596324 5025 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596327 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596329 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596332 5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596335 5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596337 5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596340 5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596342 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596345 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596349 5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596353 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596355 5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596359 5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596362 5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:03.611731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596365 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596368 5025 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596370 5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596372 5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596375 5025 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596378 5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596380 5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596383 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596385 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596388 5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596390 5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596393 5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596395 5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596398 5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596401 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596404 5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596406 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596409 5025 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596412 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596414 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:03.612278 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596417 5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596420 5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596423 5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596425 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596428 5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596431 5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596433 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596436 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596438 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596441 5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596443 5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596446 5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596449 5025 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596451 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596454 5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596456 5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596459 5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596461 5025 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596464 5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596466 5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:03.612767 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596469 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596471 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596474 5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596477 5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596480 5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596482 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596488 5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596491 5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596494 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596496 5025 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.596519 5025 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596557 5025 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596562 5025 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596565 5025 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596568 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596570 5025 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 02 06:17:03.613258 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596574 5025 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596578 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596581 5025 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596584 5025 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596587 5025 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596589 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596592 5025 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596595 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596598 5025 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596601 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596604 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596606 5025 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596609 5025 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596611 5025 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596614 5025 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596616 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596619 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596621 5025 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596624 5025 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 02 06:17:03.613650 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596626 5025 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596629 5025 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596632 5025 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596639 5025 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596642 5025 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596645 5025 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596648 5025 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596651 5025 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596655 5025 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596658 5025 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596661 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596664 5025 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596667 5025 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596670 5025 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596672 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596675 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596677 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596680 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596683 5025 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 02 06:17:03.614103 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596685 5025 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596688 5025 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596690 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596693 5025 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596696 5025 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596698 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596701 5025 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596703 5025 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596706 5025 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596708 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596711 5025 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596714 5025 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596716 5025 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596719 5025 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596721 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596724 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596731 5025 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596735 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596738 5025 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 02 06:17:03.614564 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596741 5025 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596744 5025 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596747 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596749 5025 feature_gate.go:386] unrecognized feature gate: EVPN Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596752 5025 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596755 5025 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596757 5025 feature_gate.go:386] unrecognized feature gate: Example2 Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596760 5025 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596763 5025 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596765 5025 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596768 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596771 5025 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596773 5025 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596776 5025 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596778 5025 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596781 5025 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596784 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596792 5025 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596795 5025 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596797 5025 feature_gate.go:386] unrecognized feature gate: Example Nov 02 06:17:03.615021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596800 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596803 5025 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596806 5025 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596808 5025 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596811 5025 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596813 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596816 5025 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596819 5025 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596822 5025 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596827 5025 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596830 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596833 5025 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596835 5025 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596838 5025 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596841 5025 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596843 5025 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596846 5025 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596849 5025 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596851 5025 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596854 5025 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 02 06:17:03.615685 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596857 5025 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596859 5025 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596862 5025 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:03.596865 5025 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.596916 5025 server.go:957] "Client rotation is on, will bootstrap in background" Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.597878 5025 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.597930 5025 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.598138 5025 server.go:1014] "Starting client certificate rotation" Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.598291 5025 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.598348 5025 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 06:10:51 +0000 UTC" deadline="2028-07-19 08:16:50.600816498 +0000 UTC" Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.598398 5025 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15001h59m47.002422051s" Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.598612 5025 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.600314 5025 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.602944 5025 log.go:25] "Validated CRI v1 runtime API" Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.604720 5025 log.go:25] "Validated CRI v1 image API" Nov 02 06:17:03.616417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.605462 5025 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 02 06:17:03.616944 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.607250 5025 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8397d13b-509b-48f6-b958-46b99041e4e4:/dev/nvme0n1p4 d7685f30-41ba-435d-bd41-0585f999f490:/dev/nvme0n1p3] Nov 02 06:17:03.616990 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.607266 5025 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01f0dc5e329df98637af2a43e2be37461f376413021346e2aa38421b0324b096/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01f0dc5e329df98637af2a43e2be37461f376413021346e2aa38421b0324b096/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/103a381d095d3dc8460db7dc3b9f7f1bb32c7eff5a3ae623958ad578843603b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/103a381d095d3dc8460db7dc3b9f7f1bb32c7eff5a3ae623958ad578843603b7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/234bf4a0078dbd451430f0a64a0e69814b4b8f68022a728ca2a97ae5b6b8b8a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/234bf4a0078dbd451430f0a64a0e69814b4b8f68022a728ca2a97ae5b6b8b8a3/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2954891028fe653bf4c8bfb6184c864a609ee50c0e464ab179d93495605de5dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2954891028fe653bf4c8bfb6184c864a609ee50c0e464ab179d93495605de5dc/userdata/shm major:0 minor:316 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/455abc622b558391826c4d80530a6bfdac8370de346c4ad12a140b2a3df8e060/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/455abc622b558391826c4d80530a6bfdac8370de346c4ad12a140b2a3df8e060/userdata/shm major:0 minor:228 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/472aaa6688c7b077955b4d9402776a09e43da3e06484d27f42bf129a82fe550a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/472aaa6688c7b077955b4d9402776a09e43da3e06484d27f42bf129a82fe550a/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/489ed7a9cca070d71d9dfabec5a2bcd1191e2efd5ccf43ae1e9349bb1bd763ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/489ed7a9cca070d71d9dfabec5a2bcd1191e2efd5ccf43ae1e9349bb1bd763ec/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/515cdd6d108cd258a8c18f932eee2aab666b253613d210693c976da0e42721c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/515cdd6d108cd258a8c18f932eee2aab666b253613d210693c976da0e42721c4/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/53d88e33402ed0776711a72509ab90f16abc4f74cad8462acb0570335a99b621/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/53d88e33402ed0776711a72509ab90f16abc4f74cad8462acb0570335a99b621/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54fac0e94f1ae66415d08d23dcdb220f55f5c91d40d835f010524156d691d681/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54fac0e94f1ae66415d08d23dcdb220f55f5c91d40d835f010524156d691d681/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/720138a2f707f417b09f73428a540e0cb23544375175df98084dd21057007244/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/720138a2f707f417b09f73428a540e0cb23544375175df98084dd21057007244/userdata/shm major:0 minor:197 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76a3f856faa0da3530937b29b7b962169bf14e976e7da03e6420b93b99761364/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76a3f856faa0da3530937b29b7b962169bf14e976e7da03e6420b93b99761364/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79ddf8fc3d2f1674daf4034a5fc144fed4eeb7bbf05689cf8499c8f258043156/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79ddf8fc3d2f1674daf4034a5fc144fed4eeb7bbf05689cf8499c8f258043156/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/984c9dd494487b444a38d6b0a72ef146fc1e402598de8b4f32e185c2f3b906ca/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/984c9dd494487b444a38d6b0a72ef146fc1e402598de8b4f32e185c2f3b906ca/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afe8c4db14574370b926606db8d90243b5bf4189d3df575039d31638e0272024/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afe8c4db14574370b926606db8d90243b5bf4189d3df575039d31638e0272024/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcdf3f05b5f67f1c900a64f48063888b423afaf680d80124cb8a06e144af1045/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcdf3f05b5f67f1c900a64f48063888b423afaf680d80124cb8a06e144af1045/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cce6f0dcf89ed25853c811b5a966b3dbb03a7cb0e97fe2f57dce90002c39a066/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cce6f0dcf89ed25853c811b5a966b3dbb03a7cb0e97fe2f57dce90002c39a066/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3ce2ea3476675e0f61cf5abcdb5c2eb572cc98c5a6ee422cbf9afb08427dc41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3ce2ea3476675e0f61cf5abcdb5c2eb572cc98c5a6ee422cbf9afb08427dc41/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd00b934f62eaf7987253945786fd750109f151bc6ddc3af9308ab1a0b45915d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd00b934f62eaf7987253945786fd750109f151bc6ddc3af9308ab1a0b45915d/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/de292ec76e3378e0fe776106c92c847f2c77c16c288bf83db72f157011d745e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/de292ec76e3378e0fe776106c92c847f2c77c16c288bf83db72f157011d745e8/userdata/shm major:0 minor:311 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0122975d-2172-4e2a-9af8-47f972bc5ecc/volumes/kubernetes.io~projected/kube-api-access-42bxk:{mountpoint:/var/lib/kubelet/pods/0122975d-2172-4e2a-9af8-47f972bc5ecc/volumes/kubernetes.io~projected/kube-api-access-42bxk major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0122975d-2172-4e2a-9af8-47f972bc5ecc/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0122975d-2172-4e2a-9af8-47f972bc5ecc/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0122975d-2172-4e2a-9af8-47f972bc5ecc/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/0122975d-2172-4e2a-9af8-47f972bc5ecc/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/08d074a5-dadb-4e71-a67f-9bc5c33d8857/volumes/kubernetes.io~projected/kube-api-access-ghc4z:{mountpoint:/var/lib/kubelet/pods/08d074a5-dadb-4e71-a67f-9bc5c33d8857/volumes/kubernetes.io~projected/kube-api-access-ghc4z major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf/volumes/kubernetes.io~projected/kube-api-access-sw9sf:{mountpoint:/var/lib/kubelet/pods/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf/volumes/kubernetes.io~projected/kube-api-access-sw9sf major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e/volumes/kubernetes.io~projected/kube-api-access-hxbtw:{mountpoint:/var/lib/kubelet/pods/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e/volumes/kubernetes.io~projected/kube-api-access-hxbtw major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e/volumes/kubernetes.io~secret/cert major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3100b5d8-88b0-45c8-8e06-d758f2b769fa/volumes/kubernetes.io~projected/kube-api-access-24dkw:{mountpoint:/var/lib/kubelet/pods/3100b5d8-88b0-45c8-8e06-d758f2b769fa/volumes/kubernetes.io~projected/kube-api-access-24dkw major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33574457-8ee3-4f93-80b7-3e961da45892/volumes/kubernetes.io~projected/kube-api-access-p667p:{mountpoint:/var/lib/kubelet/pods/33574457-8ee3-4f93-80b7-3e961da45892/volumes/kubernetes.io~projected/kube-api-access-p667p major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33574457-8ee3-4f93-80b7-3e961da45892/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/33574457-8ee3-4f93-80b7-3e961da45892/volumes/kubernetes.io~secret/metrics-certs major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/513b68de-5cb3-4038-9592-5eb8989af256/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/513b68de-5cb3-4038-9592-5eb8989af256/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/513b68de-5cb3-4038-9592-5eb8989af256/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/513b68de-5cb3-4038-9592-5eb8989af256/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/513b68de-5cb3-4038-9592-5eb8989af256/volumes/kubernetes.io~projected/kube-api-access-96tbp:{mountpoint:/var/lib/kubelet/pods/513b68de-5cb3-4038-9592-5eb8989af256/volumes/kubernetes.io~projected/kube-api-access-96tbp major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~projected/kube-api-access-7lq9q:{mountpoint:/var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~projected/kube-api-access-7lq9q major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~projected/registry-tls major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6ab6077a-6df9-4de9-8632-98830b9a8653/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6ab6077a-6df9-4de9-8632-98830b9a8653/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70e61026-bb17-490c-aed0-c7387729c9ef/volumes/kubernetes.io~projected/kube-api-access-jkwvv:{mountpoint:/var/lib/kubelet/pods/70e61026-bb17-490c-aed0-c7387729c9ef/volumes/kubernetes.io~projected/kube-api-access-jkwvv major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/720c25e0-1117-4a48-af60-9022daed412a/volumes/kubernetes.io~projected/kube-api-access-97d22:{mountpoint:/var/lib/kubelet/pods/720c25e0-1117-4a48-af60-9022daed412a/volumes/kubernetes.io~projected/kube-api-access-97d22 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/720c25e0-1117-4a48-af60-9022daed412a/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/720c25e0-1117-4a48-af60-9022daed412a/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/720c25e0-1117-4a48-af60-9022daed412a/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/720c25e0-1117-4a48-af60-9022daed412a/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aed3a242-64df-41a6-98b3-e2a6543918fb/volumes/kubernetes.io~projected/kube-api-access-6tsjq:{mountpoint:/var/lib/kubelet/pods/aed3a242-64df-41a6-98b3-e2a6543918fb/volumes/kubernetes.io~projected/kube-api-access-6tsjq major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba1e4bef-666b-4088-8e2b-9f7a55ab1327/volumes/kubernetes.io~projected/kube-api-access-dc4fn:{mountpoint:/var/lib/kubelet/pods/ba1e4bef-666b-4088-8e2b-9f7a55ab1327/volumes/kubernetes.io~projected/kube-api-access-dc4fn major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba1e4bef-666b-4088-8e2b-9f7a55ab1327/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/ba1e4bef-666b-4088-8e2b-9f7a55ab1327/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1842ed7-0d43-4f3d-815b-a2f45026a569/volumes/kubernetes.io~projected/kube-api-access-4nxgs:{mountpoint:/var/lib/kubelet/pods/d1842ed7-0d43-4f3d-815b-a2f45026a569/volumes/kubernetes.io~projected/kube-api-access-4nxgs major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1edfb6b-cbff-47e6-aab2-a942731747b8/volumes/kubernetes.io~projected/kube-api-access-dw58c:{mountpoint:/var/lib/kubelet/pods/e1edfb6b-cbff-47e6-aab2-a942731747b8/volumes/kubernetes.io~projected/kube-api-access-dw58c major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef/volumes/kubernetes.io~projected/kube-api-access-zljkr:{mountpoint:/var/lib/kubelet/pods/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef/volumes/kubernetes.io~projected/kube-api-access-zljkr major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f129d6cd-acad-478c-a2a7-cfa37d4c39f9/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f129d6cd-acad-478c-a2a7-cfa37d4c39f9/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f129d6cd-acad-478c-a2a7-cfa37d4c39f9/volumes/kubernetes.io~projected/kube-api-access-9zc6b:{mountpoint:/var/lib/kubelet/pods/f129d6cd-acad-478c-a2a7-cfa37d4c39f9/volumes/kubernetes.io~projected/kube-api-access-9zc6b major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f129d6cd-acad-478c-a2a7-cfa37d4c39f9/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f129d6cd-acad-478c-a2a7-cfa37d4c39f9/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc512a04-0325-41ec-b022-010c44a7c96d/volumes/kubernetes.io~projected/kube-api-access-qmcw6:{mountpoint:/var/lib/kubelet/pods/fc512a04-0325-41ec-b022-010c44a7c96d/volumes/kubernetes.io~projected/kube-api-access-qmcw6 major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fc512a04-0325-41ec-b022-010c44a7c96d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/fc512a04-0325-41ec-b022-010c44a7c96d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:218 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5443d1886f8ddfc06eb0f449d02b71bdfd2620eb002f8e43059a4b0c96051fc3/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/26aec47a89069005fd5d488b8b0e88fd092c8264bd50d9c6e20a37be23f4bd14/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7bd90c7ade63ecef8691fbf72f42968c48384f5de8bcbdcdfd77d2df3cbe6901/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/319541e6ec5d3c9dd6d8a8a8ea0cea793e42cf543bef9fb2e379bd64e1dd48eb/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/bee058c4cffe9759d73a848ac8c8bc5821b2d74b42873603bae33434e446ecc6/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/226df10a56346aeb3e5d54dbd158966a8bfe6d1aca0665440732e2b62d69abc1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/843e60591c0816583d8160f35b3f4d2e6ed10bafa18178cb41eebb80fe9a378a/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ffb77104e70f685a1d19a2512f580fc980b20594870a45513b18d334f2541529/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/8becae88d6c9f2ee095072383b4876f548c198ec6a109aa24b16a935855fa6dc/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/359e84accad081190f8fb6f04f4e38d8208db728ef94f7e9455e2b5850458e67/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/215ee145583b3b4affe9891535159dd23cd52140cf1a2713fa94f827e7b0a3b1/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/dc4271d3bb850b27f763a7ecd4f6bb5986d072cfce3d9598b2fba007b0bcab00/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-151:{mountpoint:/var/lib/containers/storage/overlay/db51516b9883436106dd6ee1d70015d75dc4d8194e7f38c9ff92fcdc9608246a/merged major:0 minor:151 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/706148dadcedc47657db9bd10ccdae63957831bb5d426ff49cf0d4352c6870ba/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3373c0c03c36e03de6df849b0b117eeeffb3b2a8b709f733dbca7e4e064b6c87/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/e5896db791e730e21a1e8f0bdf16bcb7adcce300aa51ccfc1325dfbdbe96183b/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/85c0f7c279c33893d3602ea6a4c2b57e38e0a3c465234db1d121b3e200d15d20/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/bd7012bd4641ad46c94cddb50a082e8c28bfb094ef2ac8103f8f1df2dc405538/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/455710a753267bb92e0bed6fb99afb256db883553b1b89ea042109dee3e95a28/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/20c2f32e85a967072897a92f4ca557f1afc14e6e911b7a2c02f9703d9af93ea0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-188:{mountpoint:/var/lib/containers/storage/overlay/e5dc0f85ee2ee5fe5c83158788595e19c59f7f09b02b8914909be720ddbdf64c/merged major:0 minor:188 fsType:overlay blockSize:0} overlay_0-202:{mountpoint:/var/lib/containers/storage/overlay/9261b741fc6ff6f16d6e2b7a663ae4a76cf243f21f6cdd54fda688e3e6362972/merged major:0 minor:202 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/43a39314c5c248b5591366af6ddc64d124409a85c349e8c72641c5eaa7a375c4/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/053741a35db82df5f2bb496f88aaac960bfe81a2ae47d60aa1beef4f50cc8738/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/d4b628cdfce16bf32ef48b8b87312e5d2a04e1f7630b11661ef3f3007ccb5275/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/cd492d1611c7e8d87af77cdf671d242d9788ce0b08b8fbafe77cd3ef7ce5b1db/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/1deeda235a966a1c165ae74af71a333ced89adeca9eeb992f3b66104b11e3ccf/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/58c455f0f23d122f88bd247562148b993034db2c549372e2db5a0a25fd42c8f8/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/3d75eae96d83cd3078f33d06b0b90b44fefbe9a09722544b98ceee6c9c6fc505/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/d0db10d7d16130eecdc9c2752cf8b044a5faeb12046cd1b06dc4c3ee9ecbca6e/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7221090d13705a06f86e3cf60fbb2259843f259bd6a7802fc40fcb89a76ffb30/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/dd0ac3045247ac254581f887723afb5a54be8d670f27b44a556666ea8abec2c5/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/c3e57a2b07bd183ebfcccd53a335facc30e012037bddca48fc372600e5763cc1/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/0b351df7afc3c26388aaed960ef9013842c671c7c2ffff5618c1ede48f5a0ac7/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/4abf5af91b4cb5c918c3958735113c4ac7ba68bd3f9179dc26a4718bdc4ba120/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/b49fe58731c2d4f69d31bd567ff06cfcbdd682cffb75dd8c691c10a40e64422f/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/ae3a3bb125dbca33137513557dcc4780a5698071b070e6456e4815e3d0a95682/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/09ffd9b75036a4f3c7a7cdac7c24052f7f30cb57be8a0abc35c51c67821a6bb0/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/9af100757b1790f06325d6492502f2d7a59dbd444454a684bc9083b4d6727628/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/0f180e371cdc3918b7f73474efdc5af7c29e4fe50be24d40a0658d1740507599/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/171148e3bbe8d13d8ed28df6866cbd840420d25c6e0db5dafe4a1ea4699d2f62/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/971771a6c7e06b2a0e676f657d0b72dbcb484419b0151483b9194ce72355acad/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/273c6df553f23f9776570f4246bec49c353140f03bcd6784323a15be3906009c/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/04e630c2a7fece7871378099fd5973640785427613cb74a1b65da0df12b29a29/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/97399e4167f505c5e8214e64402a9576b001c395bdaaa52c599a26b70d97c30b/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/9b825d301556729870ae36d0a4e092e17b69b573e84a1cf5aee242fcfe133cb7/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/19bde8fdc832d9425a7a6551c8e0aefe10989844a541851ea000861af20488fc/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/a62c59f600a96c44373c1f572e8e1e6c66a085ceccf7cafe437fe4c97576ef2a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/dd2adcbc0d14912812da7358370b29ab65b96c65ff94939710056ee109f2e2dd/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/98c0b6dbe4c7baddf8bc407b5f6aa0c1ec303318b039b9c6bb1f5cd9978403bd/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e8581c7fcbae3320587a777612c949c95bdbfc120a510396f6a9ccbbecca901e/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/3bea7e54f8aad12f27774ba6e330939f962bd97afaed6a7f36e2a6ca770693f3/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/e24ec9a5a006cddadfe704bc0554077b81831e9805a1dfacbe2437d989bb8505/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/f6c61dc73474ac1a3eb55ec3e25bfffa1b025a0d1aaf490f1ed1a386be3994f6/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/3c33d4d153a50ec729922c9bb56ea05ca0742a84b9f466d6e55dbf2b5622176f/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.613833 5025 manager.go:217] Machine: {Timestamp:2026-11-02 06:17:03.613384456 +0000 UTC m=+0.061761203 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100380 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b39ea8dcee5b2e8ff8e6249a4b871 SystemUUID:ec2b39ea-8dce-e5b2-e8ff-8e6249a4b871 BootID:93ac4e0a-77f8-4e35-95d8-1698722f20e3 Filesystems:[{Device:/var/lib/kubelet/pods/ba1e4bef-666b-4088-8e2b-9f7a55ab1327/volumes/kubernetes.io~projected/kube-api-access-dc4fn DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e1edfb6b-cbff-47e6-aab2-a942731747b8/volumes/kubernetes.io~projected/kube-api-access-dw58c DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70e61026-bb17-490c-aed0-c7387729c9ef/volumes/kubernetes.io~projected/kube-api-access-jkwvv DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2954891028fe653bf4c8bfb6184c864a609ee50c0e464ab179d93495605de5dc/userdata/shm DeviceMajor:0 DeviceMinor:316 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cce6f0dcf89ed25853c811b5a966b3dbb03a7cb0e97fe2f57dce90002c39a066/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afe8c4db14574370b926606db8d90243b5bf4189d3df575039d31638e0272024/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/472aaa6688c7b077955b4d9402776a09e43da3e06484d27f42bf129a82fe550a/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/455abc622b558391826c4d80530a6bfdac8370de346c4ad12a140b2a3df8e060/userdata/shm DeviceMajor:0 DeviceMinor:228 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/6ab6077a-6df9-4de9-8632-98830b9a8653/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:283 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/720c25e0-1117-4a48-af60-9022daed412a/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:305 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3100b5d8-88b0-45c8-8e06-d758f2b769fa/volumes/kubernetes.io~projected/kube-api-access-24dkw DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-202 DeviceMajor:0 DeviceMinor:202 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0122975d-2172-4e2a-9af8-47f972bc5ecc/volumes/kubernetes.io~projected/kube-api-access-42bxk DeviceMajor:0 DeviceMinor:315 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33574457-8ee3-4f93-80b7-3e961da45892/volumes/kubernetes.io~projected/kube-api-access-p667p DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/720138a2f707f417b09f73428a540e0cb23544375175df98084dd21057007244/userdata/shm DeviceMajor:0 DeviceMinor:197 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/513b68de-5cb3-4038-9592-5eb8989af256/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e/volumes/kubernetes.io~projected/kube-api-access-hxbtw DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/53d88e33402ed0776711a72509ab90f16abc4f74cad8462acb0570335a99b621/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/234bf4a0078dbd451430f0a64a0e69814b4b8f68022a728ca2a97ae5b6b8b8a3/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f129d6cd-acad-478c-a2a7-cfa37d4c39f9/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd00b934f62eaf7987253945786fd750109f151bc6ddc3af9308ab1a0b45915d/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f129d6cd-acad-478c-a2a7-cfa37d4c39f9/volumes/kubernetes.io~projected/kube-api-access-9zc6b DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ba1e4bef-666b-4088-8e2b-9f7a55ab1327/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1842ed7-0d43-4f3d-815b-a2f45026a569/volumes/kubernetes.io~projected/kube-api-access-4nxgs DeviceMajor:0 DeviceMinor:213 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/33574457-8ee3-4f93-80b7-3e961da45892/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/563834d8-a7ee-490b-a123-7358961378a2/volumes/kubernetes.io~projected/kube-api-access-7lq9q DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3ce2ea3476675e0f61cf5abcdb5c2eb572cc98c5a6ee422cbf9afb08427dc41/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/515cdd6d108cd258a8c18f932eee2aab666b253613d210693c976da0e42721c4/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f129d6cd-acad-478c-a2a7-cfa37d4c39f9/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/720c25e0-1117-4a48-af60-9022daed412a/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:309 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/de292ec76e3378e0fe776106c92c847f2c77c16c288bf83db72f157011d745e8/userdata/shm DeviceMajor:0 DeviceMinor:311 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79ddf8fc3d2f1674daf4034a5fc144fed4eeb7bbf05689cf8499c8f258043156/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/489ed7a9cca070d71d9dfabec5a2bcd1191e2efd5ccf43ae1e9349bb1bd763ec/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54fac0e94f1ae66415d08d23dcdb220f55f5c91d40d835f010524156d691d681/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc512a04-0325-41ec-b022-010c44a7c96d/volumes/kubernetes.io~projected/kube-api-access-qmcw6 DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fc512a04-0325-41ec-b022-010c44a7c96d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/720c25e0-1117-4a48-af60-9022daed412a/volumes/kubernetes.io~projected/kube-api-access-97d22 DeviceMajor:0 DeviceMinor:310 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01f0dc5e329df98637af2a43e2be37461f376413021346e2aa38421b0324b096/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0122975d-2172-4e2a-9af8-47f972bc5ecc/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:313 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-188 DeviceMajor:0 DeviceMinor:188 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef/volumes/kubernetes.io~projected/kube-api-access-zljkr DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-151 DeviceMajor:0 DeviceMinor:151 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/513b68de-5cb3-4038-9592-5eb8989af256/volumes/kubernetes.io~projected/kube-api-access-96tbp DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/513b68de-5cb3-4038-9592-5eb8989af256/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/08d074a5-dadb-4e71-a67f-9bc5c33d8857/volumes/kubernetes.io~projected/kube-api-access-ghc4z DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcdf3f05b5f67f1c900a64f48063888b423afaf680d80124cb8a06e144af1045/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf/volumes/kubernetes.io~projected/kube-api-access-sw9sf DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76a3f856faa0da3530937b29b7b962169bf14e976e7da03e6420b93b99761364/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aed3a242-64df-41a6-98b3-e2a6543918fb/volumes/kubernetes.io~projected/kube-api-access-6tsjq DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0122975d-2172-4e2a-9af8-47f972bc5ecc/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:314 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/984c9dd494487b444a38d6b0a72ef146fc1e402598de8b4f32e185c2f3b906ca/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/103a381d095d3dc8460db7dc3b9f7f1bb32c7eff5a3ae623958ad578843603b7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:234bf4a0078dbd4 MacAddress:62:04:1f:be:9b:26 Speed:10000 Mtu:8901} {Name:455abc622b55839 MacAddress:f6:35:b6:9a:58:a0 Speed:10000 Mtu:8901} {Name:472aaa6688c7b07 MacAddress:5e:9f:8e:cb:75:de Speed:10000 Mtu:8901} {Name:489ed7a9cca070d MacAddress:46:71:5c:a6:83:81 Speed:10000 Mtu:8901} {Name:720138a2f707f41 MacAddress:2a:7a:70:be:c9:87 Speed:10000 Mtu:8901} {Name:76a3f856faa0da3 MacAddress:72:4a:93:18:12:ab Speed:10000 Mtu:8901} {Name:79ddf8fc3d2f167 MacAddress:ae:4c:e9:7c:3c:be Speed:10000 Mtu:8901} {Name:bcdf3f05b5f67f1 MacAddress:26:4e:63:dd:b7:11 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:e3:0c:b7:69 Speed:0 Mtu:9001} {Name:br-int MacAddress:0e:c4:c2:a9:2c:89 Speed:0 Mtu:8901} {Name:de292ec76e3378e MacAddress:d2:2b:28:76:4b:22 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:e3:0c:b7:69 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:b6:a4:83:3b:ac:0e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:1a:a5:9c:c4:85 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614132 5025 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614173 5025 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614426 5025 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614440 5025 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-34.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614579 5025 topology_manager.go:143] "Creating topology manager with none policy" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614587 5025 container_manager_linux.go:311] "Creating device plugin manager" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614597 5025 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614609 5025 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614684 5025 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614833 5025 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614932 5025 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614964 5025 kubelet.go:493] "Attempting to sync node with API server" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614973 5025 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614982 5025 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.614992 5025 kubelet.go:405] "Adding apiserver pod source" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.615000 5025 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.615665 5025 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.615680 5025 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:17:03.617211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.616555 5025 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 02 06:17:03.622986 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.622971 5025 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 02 06:17:03.622986 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.622985 5025 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 02 06:17:03.623176 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623114 5025 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 02 06:17:03.623325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623312 5025 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 02 06:17:03.623361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623345 5025 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 02 06:17:03.623466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623453 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 02 06:17:03.623513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623477 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 02 06:17:03.623513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623489 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 02 06:17:03.623513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623500 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 02 06:17:03.623513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623509 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 02 06:17:03.623620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623515 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 02 06:17:03.623620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623530 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 02 06:17:03.623620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623535 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 02 06:17:03.623620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623543 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 02 06:17:03.623620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623549 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 02 06:17:03.623620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623569 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 02 06:17:03.623620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623580 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 02 06:17:03.623904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623633 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 02 06:17:03.623904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.623642 5025 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 02 06:17:03.627227 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.627212 5025 server.go:1263] "Started kubelet" Nov 02 06:17:03.627329 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.627257 5025 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 02 06:17:03.627437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.627397 5025 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 02 06:17:03.630061 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.627771 5025 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 02 06:17:03.630061 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.628743 5025 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 02 06:17:03.628033 ip-10-0-130-34 systemd[1]: Started Kubernetes Kubelet. Nov 02 06:17:03.631405 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.631385 5025 server.go:317] "Adding debug handlers to kubelet server" Nov 02 06:17:03.635478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.635130 5025 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 02 06:17:03.635478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.635161 5025 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 02 06:17:03.635478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.635233 5025 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 06:10:52 +0000 UTC" deadline="2028-06-12 06:37:19.699579219 +0000 UTC" Nov 02 06:17:03.635478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.635256 5025 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14112h20m16.064326812s" Nov 02 06:17:03.635478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.635275 5025 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 02 06:17:03.635478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.635286 5025 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 02 06:17:03.635478 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:03.635318 5025 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-34.us-west-2.compute.internal\" not found" Nov 02 06:17:03.635478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.635338 5025 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 02 06:17:03.637355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637293 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15fd4393-b2d4-4f12-8ae8-f38b92a3b05e" volumeName="kubernetes.io/secret/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-cert" seLinuxMountContext="" Nov 02 06:17:03.637355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637348 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="513b68de-5cb3-4038-9592-5eb8989af256" volumeName="kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-etc-tuned" seLinuxMountContext="" Nov 02 06:17:03.637355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637363 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="513b68de-5cb3-4038-9592-5eb8989af256" volumeName="kubernetes.io/projected/513b68de-5cb3-4038-9592-5eb8989af256-kube-api-access-96tbp" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637375 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad" volumeName="kubernetes.io/secret/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-agent-certs" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637389 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1edfb6b-cbff-47e6-aab2-a942731747b8" volumeName="kubernetes.io/empty-dir/e1edfb6b-cbff-47e6-aab2-a942731747b8-tmp-dir" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637402 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33574457-8ee3-4f93-80b7-3e961da45892" volumeName="kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637414 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="513b68de-5cb3-4038-9592-5eb8989af256" volumeName="kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-tmp" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637426 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="563834d8-a7ee-490b-a123-7358961378a2" volumeName="kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-kube-api-access-7lq9q" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637443 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aed3a242-64df-41a6-98b3-e2a6543918fb" volumeName="kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637465 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="15fd4393-b2d4-4f12-8ae8-f38b92a3b05e" volumeName="kubernetes.io/projected/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-kube-api-access-hxbtw" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637499 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aed3a242-64df-41a6-98b3-e2a6543918fb" volumeName="kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-sysctl-allowlist" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637523 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f129d6cd-acad-478c-a2a7-cfa37d4c39f9" volumeName="kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-config" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637535 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f129d6cd-acad-478c-a2a7-cfa37d4c39f9" volumeName="kubernetes.io/secret/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovn-node-metrics-cert" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637546 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="563834d8-a7ee-490b-a123-7358961378a2" volumeName="kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-bound-sa-token" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637561 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0122975d-2172-4e2a-9af8-47f972bc5ecc" volumeName="kubernetes.io/empty-dir/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-textfile" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637573 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11477d6f-1bd9-40c9-8d57-c0a0e1edaebf" volumeName="kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cni-binary-copy" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637584 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33574457-8ee3-4f93-80b7-3e961da45892" volumeName="kubernetes.io/projected/33574457-8ee3-4f93-80b7-3e961da45892-kube-api-access-p667p" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637600 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="720c25e0-1117-4a48-af60-9022daed412a" volumeName="kubernetes.io/projected/720c25e0-1117-4a48-af60-9022daed412a-kube-api-access-97d22" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637620 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aed3a242-64df-41a6-98b3-e2a6543918fb" volumeName="kubernetes.io/projected/aed3a242-64df-41a6-98b3-e2a6543918fb-kube-api-access-6tsjq" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637631 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc512a04-0325-41ec-b022-010c44a7c96d" volumeName="kubernetes.io/configmap/fc512a04-0325-41ec-b022-010c44a7c96d-config-volume" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637645 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f129d6cd-acad-478c-a2a7-cfa37d4c39f9" volumeName="kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-env-overrides" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637660 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0122975d-2172-4e2a-9af8-47f972bc5ecc" volumeName="kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-metrics-client-ca" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637672 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11477d6f-1bd9-40c9-8d57-c0a0e1edaebf" volumeName="kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-daemon-config" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637686 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e61026-bb17-490c-aed0-c7387729c9ef" volumeName="kubernetes.io/configmap/70e61026-bb17-490c-aed0-c7387729c9ef-serviceca" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637699 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aed3a242-64df-41a6-98b3-e2a6543918fb" volumeName="kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-binary-copy" seLinuxMountContext="" Nov 02 06:17:03.637702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637714 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba1e4bef-666b-4088-8e2b-9f7a55ab1327" volumeName="kubernetes.io/configmap/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637727 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11477d6f-1bd9-40c9-8d57-c0a0e1edaebf" volumeName="kubernetes.io/projected/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-kube-api-access-sw9sf" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637738 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1842ed7-0d43-4f3d-815b-a2f45026a569" volumeName="kubernetes.io/projected/d1842ed7-0d43-4f3d-815b-a2f45026a569-kube-api-access-4nxgs" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637753 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec8433d1-0c6c-4732-b8a4-d73c251cd5ef" volumeName="kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637811 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3100b5d8-88b0-45c8-8e06-d758f2b769fa" volumeName="kubernetes.io/projected/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kube-api-access-24dkw" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637824 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="720c25e0-1117-4a48-af60-9022daed412a" volumeName="kubernetes.io/configmap/720c25e0-1117-4a48-af60-9022daed412a-metrics-client-ca" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637836 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc512a04-0325-41ec-b022-010c44a7c96d" volumeName="kubernetes.io/empty-dir/fc512a04-0325-41ec-b022-010c44a7c96d-tmp-dir" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637850 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0122975d-2172-4e2a-9af8-47f972bc5ecc" volumeName="kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-tls" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637877 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08d074a5-dadb-4e71-a67f-9bc5c33d8857" volumeName="kubernetes.io/configmap/08d074a5-dadb-4e71-a67f-9bc5c33d8857-iptables-alerter-script" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637888 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="563834d8-a7ee-490b-a123-7358961378a2" volumeName="kubernetes.io/empty-dir/563834d8-a7ee-490b-a123-7358961378a2-ca-trust-extracted" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637901 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba1e4bef-666b-4088-8e2b-9f7a55ab1327" volumeName="kubernetes.io/projected/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-api-access-dc4fn" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637912 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="563834d8-a7ee-490b-a123-7358961378a2" volumeName="kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-trusted-ca" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637924 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="563834d8-a7ee-490b-a123-7358961378a2" volumeName="kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-installation-pull-secrets" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637938 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6ab6077a-6df9-4de9-8632-98830b9a8653" volumeName="kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637949 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70e61026-bb17-490c-aed0-c7387729c9ef" volumeName="kubernetes.io/projected/70e61026-bb17-490c-aed0-c7387729c9ef-kube-api-access-jkwvv" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637956 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="720c25e0-1117-4a48-af60-9022daed412a" volumeName="kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-tls" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637965 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="563834d8-a7ee-490b-a123-7358961378a2" volumeName="kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-image-registry-private-configuration" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637972 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0122975d-2172-4e2a-9af8-47f972bc5ecc" volumeName="kubernetes.io/projected/0122975d-2172-4e2a-9af8-47f972bc5ecc-kube-api-access-42bxk" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637980 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="08d074a5-dadb-4e71-a67f-9bc5c33d8857" volumeName="kubernetes.io/projected/08d074a5-dadb-4e71-a67f-9bc5c33d8857-kube-api-access-ghc4z" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637990 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="563834d8-a7ee-490b-a123-7358961378a2" volumeName="kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-registry-tls" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.637998 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad" volumeName="kubernetes.io/configmap/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-konnectivity-ca" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638006 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba1e4bef-666b-4088-8e2b-9f7a55ab1327" volumeName="kubernetes.io/empty-dir/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-data-volume" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638015 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f129d6cd-acad-478c-a2a7-cfa37d4c39f9" volumeName="kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-script-lib" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638022 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0122975d-2172-4e2a-9af8-47f972bc5ecc" volumeName="kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638030 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="563834d8-a7ee-490b-a123-7358961378a2" volumeName="kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-registry-certificates" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638036 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f129d6cd-acad-478c-a2a7-cfa37d4c39f9" volumeName="kubernetes.io/projected/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-kube-api-access-9zc6b" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638044 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc512a04-0325-41ec-b022-010c44a7c96d" volumeName="kubernetes.io/projected/fc512a04-0325-41ec-b022-010c44a7c96d-kube-api-access-qmcw6" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638051 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba1e4bef-666b-4088-8e2b-9f7a55ab1327" volumeName="kubernetes.io/secret/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638063 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="720c25e0-1117-4a48-af60-9022daed412a" volumeName="kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638123 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1edfb6b-cbff-47e6-aab2-a942731747b8" volumeName="kubernetes.io/projected/e1edfb6b-cbff-47e6-aab2-a942731747b8-kube-api-access-dw58c" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638132 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0122975d-2172-4e2a-9af8-47f972bc5ecc" volumeName="kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638164 5025 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fc512a04-0325-41ec-b022-010c44a7c96d" volumeName="kubernetes.io/secret/fc512a04-0325-41ec-b022-010c44a7c96d-metrics-tls" seLinuxMountContext="" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638172 5025 reconstruct.go:97] "Volume reconstruction finished" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638176 5025 reconciler.go:29] "Reconciler: start to sync state" Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638510 5025 factory.go:153] Registering CRI-O factory Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638528 5025 factory.go:223] Registration of the crio container factory successfully Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638609 5025 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638619 5025 factory.go:55] Registering systemd factory Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638629 5025 factory.go:223] Registration of the systemd container factory successfully Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638647 5025 factory.go:103] Registering Raw factory Nov 02 06:17:03.638866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638663 5025 manager.go:1199] Started watching for new ooms in manager Nov 02 06:17:03.640704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.638942 5025 manager.go:319] Starting recovery of all containers Nov 02 06:17:03.659505 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.659456 5025 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 02 06:17:03.661025 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.661001 5025 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 02 06:17:03.661163 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.661030 5025 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 02 06:17:03.661163 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.661056 5025 kubelet.go:2521] "Starting kubelet main sync loop" Nov 02 06:17:03.661163 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:03.661118 5025 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 02 06:17:03.695996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.695963 5025 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="6a9da6ac149a0047caccb04cc43d21e54c7e2b6616286972590b06b293574459" exitCode=0 Nov 02 06:17:03.695996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.695990 5025 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="1b911b79378594ea3f5a1d7f97b1d425574912ad1157d71b7030e792e8f7add1" exitCode=0 Nov 02 06:17:03.695996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.695999 5025 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="e9e19b2cfc554d7e00e5dbae24e934bd03c91fbeec43c90c96e4aea47dc5639c" exitCode=0 Nov 02 06:17:03.696288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.696009 5025 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="d40959ac1049e22ac8752cbd05c9cf457ddb54f16b8886bb5dd366e348c7580a" exitCode=0 Nov 02 06:17:03.696288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.696018 5025 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="3eddefc7360763a88ff20dc569bb110ad190a0cdc37bed7ea1e0b1e1a0cc1b5b" exitCode=0 Nov 02 06:17:03.696288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.696024 5025 generic.go:356] "Generic (PLEG): container finished" podID="aed3a242-64df-41a6-98b3-e2a6543918fb" containerID="977eb3b23839066d993507f6eadbc2884b4e168ea1de4644f4c01e1fb1e95457" exitCode=0 Nov 02 06:17:03.709972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.709940 5025 generic.go:356] "Generic (PLEG): container finished" podID="b2c68cd55c462db386dbcd1795bbdd95" containerID="a33b745b35d223959b65ba8bfaa6ac4bd7c23b2effed05921ec421101747eb2b" exitCode=0 Nov 02 06:17:03.732613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.732585 5025 manager.go:324] Recovery completed Nov 02 06:17:03.735445 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:03.735385 5025 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-34.us-west-2.compute.internal\" not found" Nov 02 06:17:03.748017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.747996 5025 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:17:03.749850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.749834 5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:17:03.749949 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.749862 5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:17:03.749949 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.749873 5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:17:03.752238 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752221 5025 cpu_manager.go:226] "Starting" policy="none" Nov 02 06:17:03.752238 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752233 5025 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:17:03.752379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752234 5025 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 02 06:17:03.752379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752342 5025 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 02 06:17:03.752532 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752513 5025 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 02 06:17:03.752605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752529 5025 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 02 06:17:03.752605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752563 5025 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 02 06:17:03.752605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752572 5025 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 02 06:17:03.752605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752581 5025 policy_none.go:50] "Start" Nov 02 06:17:03.752605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752595 5025 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 02 06:17:03.752832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752609 5025 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 02 06:17:03.752832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752789 5025 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 02 06:17:03.752832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752802 5025 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 02 06:17:03.752832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.752810 5025 policy_none.go:44] "Start" Nov 02 06:17:03.753052 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.753002 5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:17:03.753052 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.753026 5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:17:03.753052 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.753036 5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:17:03.757472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.757439 5025 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 02 06:17:03.761673 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:03.761651 5025 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 02 06:17:03.762362 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.762350 5025 manager.go:341] "Starting Device Plugin manager" Nov 02 06:17:03.762406 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:03.762374 5025 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 02 06:17:03.762406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.762383 5025 server.go:87] "Starting device plugin registration server" Nov 02 06:17:03.762695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.762680 5025 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 02 06:17:03.762752 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.762699 5025 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 02 06:17:03.762806 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.762787 5025 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 02 06:17:03.762937 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.762890 5025 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 02 06:17:03.762996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.762965 5025 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 02 06:17:03.762996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.762973 5025 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 02 06:17:03.763099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.763002 5025 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T06:17:03.76291718Z","UUID":"f9ac79c0-f070-4ddc-9d56-af52f3d1c5e9","Handler":null,"Name":"","Endpoint":""} Nov 02 06:17:03.764782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.764751 5025 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 02 06:17:03.764878 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:03.764855 5025 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 02 06:17:03.764934 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:03.764912 5025 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-34.us-west-2.compute.internal\" not found" Nov 02 06:17:03.764934 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.764871 5025 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 02 06:17:03.775749 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.775729 5025 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:17:03.775840 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.775754 5025 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:17:03.863642 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.863609 5025 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 02 06:17:03.864660 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.864641 5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientMemory" Nov 02 06:17:03.864756 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.864677 5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Nov 02 06:17:03.864756 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.864688 5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeHasSufficientPID" Nov 02 06:17:03.864756 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.864708 5025 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:03.869700 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.869681 5025 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 02 06:17:03.870940 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.870922 5025 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:17:03.916546 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.916516 5025 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 02 06:17:03.962203 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.962157 5025 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal"] Nov 02 06:17:03.962782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.962738 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" event={"ID":"ceea1ff964927afa3d04bca3afe81300","Type":"ContainerStarted","Data":"00fb99159cbcde00a716abba6907c68f546a94ece424f79c22dc72ff0520c19f"} Nov 02 06:17:03.962842 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.962797 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" event={"ID":"ceea1ff964927afa3d04bca3afe81300","Type":"ContainerStarted","Data":"dd00b934f62eaf7987253945786fd750109f151bc6ddc3af9308ab1a0b45915d"} Nov 02 06:17:03.962842 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.962819 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" event={"ID":"b2c68cd55c462db386dbcd1795bbdd95","Type":"ContainerStarted","Data":"ce46fb649f248e52d089caa3308babb18f287083a951347dfb1fdb2b7d94dccb"} Nov 02 06:17:03.962842 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.962839 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" event={"ID":"b2c68cd55c462db386dbcd1795bbdd95","Type":"ContainerDied","Data":"a33b745b35d223959b65ba8bfaa6ac4bd7c23b2effed05921ec421101747eb2b"} Nov 02 06:17:03.962954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.962858 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" event={"ID":"b2c68cd55c462db386dbcd1795bbdd95","Type":"ContainerStarted","Data":"54fac0e94f1ae66415d08d23dcdb220f55f5c91d40d835f010524156d691d681"} Nov 02 06:17:03.962954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.962865 5025 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:03.962954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.962943 5025 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:03.972107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.972069 5025 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:03.972217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.972152 5025 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:03.972719 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.972704 5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeNotReady" Nov 02 06:17:03.972815 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:03.972735 5025 setters.go:546] "Node became not ready" node="ip-10-0-130-34.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T06:17:03Z","lastTransitionTime":"2026-11-02T06:17:03Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 02 06:17:04.019734 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.019673 5025 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:17:04.019734 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:04.019719 5025 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.020466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.020451 5025 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:17:04.020504 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:04.020487 5025 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.025329 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.025309 5025 nodeinfomanager.go:471] existing CSINode "ip-10-0-130-34.us-west-2.compute.internal" is owned by different node (oldNodeID="093c5e12-cda3-4c7b-ba02-20b2546c4f30", newNodeID=""), cleaning up... Nov 02 06:17:04.071366 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.071337 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceea1ff964927afa3d04bca3afe81300-config\") pod \"kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"ceea1ff964927afa3d04bca3afe81300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.071546 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.071374 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.071546 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.071398 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.172239 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.172200 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.172239 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.172151 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.172442 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.172272 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceea1ff964927afa3d04bca3afe81300-config\") pod \"kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"ceea1ff964927afa3d04bca3afe81300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.172442 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.172293 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.172442 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.172332 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b2c68cd55c462db386dbcd1795bbdd95-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"b2c68cd55c462db386dbcd1795bbdd95\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.172442 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.172343 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ceea1ff964927afa3d04bca3afe81300-config\") pod \"kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal\" (UID: \"ceea1ff964927afa3d04bca3afe81300\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.615389 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.615363 5025 apiserver.go:52] "Watching apiserver" Nov 02 06:17:04.713767 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.713734 5025 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 02 06:17:04.714335 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.714301 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wvjp6","openshift-multus/network-metrics-daemon-5wc8x","openshift-image-registry/image-registry-5b4b56fff9-77j92","openshift-network-operator/iptables-alerter-8tczj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-ccb2x","kube-system/konnectivity-agent-fc9xt","kube-system/global-pull-secret-syncer-4vv5l","openshift-ingress-canary/ingress-canary-x9xwq","openshift-insights/insights-runtime-extractor-x48kz","openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc","openshift-monitoring/openshift-state-metrics-7bd7489957-p7227","openshift-multus/multus-additional-cni-plugins-gm9nl","openshift-network-diagnostics/network-check-target-k7c7h","openshift-dns/node-resolver-5jr5c","openshift-image-registry/node-ca-58ckp","openshift-multus/multus-c9z9s","kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal","openshift-monitoring/node-exporter-7wr46","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc","openshift-ovn-kubernetes/ovnkube-node-x4qlj"] Nov 02 06:17:04.721298 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.721272 5025 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.721439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.721346 5025 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.736699 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.736678 5025 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 02 06:17:04.765809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.765787 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.765809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.765787 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 02 06:17:04.765961 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.765788 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.765961 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.765842 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 02 06:17:04.766293 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.766275 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.766356 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.766301 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 02 06:17:04.766426 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.766353 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 02 06:17:04.766474 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.766451 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 02 06:17:04.766578 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.766565 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 02 06:17:04.766719 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.766703 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.767299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.767284 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.767369 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.767298 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.767369 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.767354 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.767472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.767375 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 02 06:17:04.767472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.767429 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.768756 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.768738 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.769018 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.769003 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.769056 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.769036 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.771192 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.771179 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.771438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.771427 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.772255 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.772240 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 02 06:17:04.772609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.772593 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 02 06:17:04.772667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.772622 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 02 06:17:04.772779 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.772767 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.776010 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.775992 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-cert\") pod \"ingress-canary-x9xwq\" (UID: \"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e\") " pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:17:04.776107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776022 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-config\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.776107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776049 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.776201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776171 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-textfile\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.776277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776257 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-trusted-ca\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.776337 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776280 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-textfile\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.776337 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776290 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-host\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.776337 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776310 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-node-log\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.776450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776345 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-netns\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.776450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776348 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-config\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.776450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776369 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08d074a5-dadb-4e71-a67f-9bc5c33d8857-iptables-alerter-script\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:17:04.776450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776407 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-konnectivity-ca\") pod \"konnectivity-agent-fc9xt\" (UID: \"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad\") " pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:17:04.776651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776475 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc512a04-0325-41ec-b022-010c44a7c96d-config-volume\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:17:04.776651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776552 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/08d074a5-dadb-4e71-a67f-9bc5c33d8857-iptables-alerter-script\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:17:04.776651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776583 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.776651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776618 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.776651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776632 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-konnectivity-ca\") pod \"konnectivity-agent-fc9xt\" (UID: \"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad\") " pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:17:04.776851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776653 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovn-node-metrics-cert\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.776851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776697 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-registry-tls\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.776851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776724 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-hostroot\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.776851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776752 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.776851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776831 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysconfig\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.777041 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776865 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tbp\" (UniqueName: \"kubernetes.io/projected/513b68de-5cb3-4038-9592-5eb8989af256-kube-api-access-96tbp\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.777041 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776898 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.777041 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.776985 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zc6b\" (UniqueName: \"kubernetes.io/projected/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-kube-api-access-9zc6b\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.777041 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777030 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmcw6\" (UniqueName: \"kubernetes.io/projected/fc512a04-0325-41ec-b022-010c44a7c96d-kube-api-access-qmcw6\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:17:04.777228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777148 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-sys-fs\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.777228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777197 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-log-socket\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.777314 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777239 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.777314 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777278 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:04.777406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777307 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-sys\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.777406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777334 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42bxk\" (UniqueName: \"kubernetes.io/projected/0122975d-2172-4e2a-9af8-47f972bc5ecc-kube-api-access-42bxk\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.777406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777401 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-multus-certs\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.777513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777421 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:17:04.777513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777476 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-bound-sa-token\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.777513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777494 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-script-lib\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.777513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777508 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zljkr\" (UniqueName: \"kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr\") pod \"network-check-target-k7c7h\" (UID: \"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef\") " pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:17:04.777700 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777530 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghc4z\" (UniqueName: \"kubernetes.io/projected/08d074a5-dadb-4e71-a67f-9bc5c33d8857-kube-api-access-ghc4z\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:17:04.777700 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777545 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-tls\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.777700 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777564 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-wtmp\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.777700 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777587 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-installation-pull-secrets\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.777700 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777605 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dkw\" (UniqueName: \"kubernetes.io/projected/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kube-api-access-24dkw\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.777700 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777661 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70e61026-bb17-490c-aed0-c7387729c9ef-serviceca\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:17:04.777954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777732 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-var-lib-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.777954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777772 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-kubelet-config\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:17:04.777954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777864 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-etc-tuned\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.777954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777939 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-env-overrides\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.778123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777966 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovnkube-script-lib\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.778123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777967 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70e61026-bb17-490c-aed0-c7387729c9ef-host\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:17:04.778123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.777997 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-etc-tuned\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.778123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778021 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.778123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778046 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/720c25e0-1117-4a48-af60-9022daed412a-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:04.778277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778125 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkwvv\" (UniqueName: \"kubernetes.io/projected/70e61026-bb17-490c-aed0-c7387729c9ef-kube-api-access-jkwvv\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:17:04.778277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778143 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-env-overrides\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.778277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778209 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:17:04.778277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778254 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc4fn\" (UniqueName: \"kubernetes.io/projected/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-api-access-dc4fn\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:17:04.778395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778288 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-socket-dir-parent\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.778395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778304 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-kubelet\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.778395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778320 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-k8s-cni-cncf-io\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.778395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778344 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:17:04.778395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778371 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-kubelet\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.778557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778404 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/70e61026-bb17-490c-aed0-c7387729c9ef-serviceca\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:17:04.778557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778429 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-bin\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.778557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778460 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-dbus\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:17:04.778557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778483 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-device-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.778557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778513 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-var-lib-kubelet\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.778557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778536 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc512a04-0325-41ec-b022-010c44a7c96d-tmp-dir\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:17:04.778774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778586 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc512a04-0325-41ec-b022-010c44a7c96d-tmp-dir\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:17:04.778774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778610 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-multus\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.778774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778653 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-conf-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.778774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778697 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-daemon-config\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.778929 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778811 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.778929 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778825 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-daemon-config\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.778929 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778841 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-socket-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.778929 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778887 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-etc-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.778929 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778914 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-modprobe-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.779161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778936 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-kubernetes\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.779161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778952 5025 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:17:04.779161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.778973 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1edfb6b-cbff-47e6-aab2-a942731747b8-tmp-dir\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:17:04.779161 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:04.779005 5025 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.779161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779012 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/563834d8-a7ee-490b-a123-7358961378a2-ca-trust-extracted\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.779161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779042 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-lib-modules\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.779161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779059 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1edfb6b-cbff-47e6-aab2-a942731747b8-tmp-dir\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:17:04.779161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779070 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.779161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779114 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc512a04-0325-41ec-b022-010c44a7c96d-metrics-tls\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:17:04.779161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779140 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/563834d8-a7ee-490b-a123-7358961378a2-ca-trust-extracted\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779175 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-slash\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779207 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08d074a5-dadb-4e71-a67f-9bc5c33d8857-host-slash\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779232 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-ovn\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779263 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779306 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-crio-socket\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779332 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779335 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779383 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-metrics-client-ca\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779423 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-run\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779454 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1edfb6b-cbff-47e6-aab2-a942731747b8-hosts-file\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779472 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.779511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779484 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-agent-certs\") pod \"konnectivity-agent-fc9xt\" (UID: \"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad\") " pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779532 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-netns\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779549 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779567 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-image-registry-private-configuration\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779584 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lq9q\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-kube-api-access-7lq9q\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779598 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cni-binary-copy\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779615 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw58c\" (UniqueName: \"kubernetes.io/projected/e1edfb6b-cbff-47e6-aab2-a942731747b8-kube-api-access-dw58c\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779639 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxbtw\" (UniqueName: \"kubernetes.io/projected/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-kube-api-access-hxbtw\") pod \"ingress-canary-x9xwq\" (UID: \"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e\") " pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779693 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-systemd-units\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779713 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-etc-kubernetes\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779738 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-sys\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779770 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-root\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779789 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cni-binary-copy\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779787 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aed3a242-64df-41a6-98b3-e2a6543918fb-cni-binary-copy\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779789 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-netd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779859 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tsjq\" (UniqueName: \"kubernetes.io/projected/aed3a242-64df-41a6-98b3-e2a6543918fb-kube-api-access-6tsjq\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779881 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-registry-certificates\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779898 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-cnibin\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779913 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-data-volume\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779949 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-system-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779968 5025 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 02 06:17:04.779982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779986 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p667p\" (UniqueName: \"kubernetes.io/projected/33574457-8ee3-4f93-80b7-3e961da45892-kube-api-access-p667p\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.779990 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-data-volume\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:04.780013 5025 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-34.us-west-2.compute.internal" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780035 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-systemd\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780066 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97d22\" (UniqueName: \"kubernetes.io/projected/720c25e0-1117-4a48-af60-9022daed412a-kube-api-access-97d22\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780118 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw9sf\" (UniqueName: \"kubernetes.io/projected/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-kube-api-access-sw9sf\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780144 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-system-cni-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780168 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-bin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780190 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-systemd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780210 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-registry-certificates\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780275 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780357 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-conf\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780386 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-tmp\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780403 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780430 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780448 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cnibin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780464 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-registration-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780506 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nxgs\" (UniqueName: \"kubernetes.io/projected/d1842ed7-0d43-4f3d-815b-a2f45026a569-kube-api-access-4nxgs\") pod \"migrator-66df5d496b-qx2rc\" (UID: \"d1842ed7-0d43-4f3d-815b-a2f45026a569\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc" Nov 02 06:17:04.780827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.780523 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/513b68de-5cb3-4038-9592-5eb8989af256-tmp\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.830627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.830600 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zc6b\" (UniqueName: \"kubernetes.io/projected/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-kube-api-access-9zc6b\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.832225 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.832206 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-bound-sa-token\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.832718 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.832694 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dkw\" (UniqueName: \"kubernetes.io/projected/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kube-api-access-24dkw\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.832791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.832759 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghc4z\" (UniqueName: \"kubernetes.io/projected/08d074a5-dadb-4e71-a67f-9bc5c33d8857-kube-api-access-ghc4z\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:17:04.832882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.832862 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkwvv\" (UniqueName: \"kubernetes.io/projected/70e61026-bb17-490c-aed0-c7387729c9ef-kube-api-access-jkwvv\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:17:04.832928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.832913 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmcw6\" (UniqueName: \"kubernetes.io/projected/fc512a04-0325-41ec-b022-010c44a7c96d-kube-api-access-qmcw6\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:17:04.833023 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.833008 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lq9q\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-kube-api-access-7lq9q\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.833191 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.833171 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tbp\" (UniqueName: \"kubernetes.io/projected/513b68de-5cb3-4038-9592-5eb8989af256-kube-api-access-96tbp\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.833371 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.833350 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p667p\" (UniqueName: \"kubernetes.io/projected/33574457-8ee3-4f93-80b7-3e961da45892-kube-api-access-p667p\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:17:04.833839 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.833823 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tsjq\" (UniqueName: \"kubernetes.io/projected/aed3a242-64df-41a6-98b3-e2a6543918fb-kube-api-access-6tsjq\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.833903 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.833889 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw9sf\" (UniqueName: \"kubernetes.io/projected/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-kube-api-access-sw9sf\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.834599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.834584 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw58c\" (UniqueName: \"kubernetes.io/projected/e1edfb6b-cbff-47e6-aab2-a942731747b8-kube-api-access-dw58c\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:17:04.852892 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.852863 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-9bfjz\"" Nov 02 06:17:04.852892 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.852875 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 02 06:17:04.853007 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.852863 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 02 06:17:04.853007 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.852956 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 02 06:17:04.853111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853009 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rpvjg\"" Nov 02 06:17:04.853111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853050 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 02 06:17:04.853111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853069 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 02 06:17:04.853111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853099 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rkkc6\"" Nov 02 06:17:04.853283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853129 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-cglw4\"" Nov 02 06:17:04.853283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853250 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 02 06:17:04.853362 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853330 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-d5zsv\"" Nov 02 06:17:04.853596 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853578 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rwgrc\"" Nov 02 06:17:04.853657 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853605 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 02 06:17:04.853657 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853630 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-cdk7k\"" Nov 02 06:17:04.853657 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853633 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2646z\"" Nov 02 06:17:04.853908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853888 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-flhk8\"" Nov 02 06:17:04.853908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.853889 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Nov 02 06:17:04.854119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854102 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-77l82\"" Nov 02 06:17:04.854183 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854129 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 02 06:17:04.854183 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854155 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xbcc6\"" Nov 02 06:17:04.854183 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854160 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 02 06:17:04.854183 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854169 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-75dck\"" Nov 02 06:17:04.854183 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854129 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 02 06:17:04.854380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854185 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4lqj8\"" Nov 02 06:17:04.854380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854187 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 02 06:17:04.854380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854188 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 02 06:17:04.854380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854219 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-kqjnz\"" Nov 02 06:17:04.854380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854219 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nxd68\"" Nov 02 06:17:04.854380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854129 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 02 06:17:04.857159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854631 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-d57td\"" Nov 02 06:17:04.857159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854692 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-zth7z\"" Nov 02 06:17:04.857159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854706 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Nov 02 06:17:04.857159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854777 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xffwp\"" Nov 02 06:17:04.857159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.854778 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-bpw78\"" Nov 02 06:17:04.857457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.857334 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-cert\") pod \"ingress-canary-x9xwq\" (UID: \"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e\") " pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:17:04.857457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.857375 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-ovn-node-metrics-cert\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.857946 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.857923 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/563834d8-a7ee-490b-a123-7358961378a2-registry-tls\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.858036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.857982 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:04.858117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.858059 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-tls\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.858117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.858069 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-installation-pull-secrets\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.858673 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.858657 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:17:04.858717 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.858658 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/33574457-8ee3-4f93-80b7-3e961da45892-metrics-certs\") pod \"network-metrics-daemon-5wc8x\" (UID: \"33574457-8ee3-4f93-80b7-3e961da45892\") " pod="openshift-multus/network-metrics-daemon-5wc8x" Nov 02 06:17:04.859796 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.859779 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/563834d8-a7ee-490b-a123-7358961378a2-image-registry-private-configuration\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.859796 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.859786 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc512a04-0325-41ec-b022-010c44a7c96d-metrics-tls\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:17:04.859894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.859808 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad-agent-certs\") pod \"konnectivity-agent-fc9xt\" (UID: \"b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad\") " pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:17:04.860945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.860928 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/720c25e0-1117-4a48-af60-9022daed412a-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:04.861029 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.861004 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.861066 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.861018 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6ab6077a-6df9-4de9-8632-98830b9a8653-original-pull-secret\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:17:04.865518 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.865477 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.865518 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.865506 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.865828 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.865803 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.865899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.865817 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 02 06:17:04.865899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.865834 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 02 06:17:04.865899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.865841 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.865899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.865853 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.865899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.865866 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 02 06:17:04.866106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.865908 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 02 06:17:04.866106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.865908 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.866106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.865982 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.866106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.866050 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.866267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.866191 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 02 06:17:04.867248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.867232 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc512a04-0325-41ec-b022-010c44a7c96d-config-volume\") pod \"dns-default-wvjp6\" (UID: \"fc512a04-0325-41ec-b022-010c44a7c96d\") " pod="openshift-dns/dns-default-wvjp6" Nov 02 06:17:04.867325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.867310 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-accelerators-collector-config\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.868426 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.868410 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:17:04.868653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.868639 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nxgs\" (UniqueName: \"kubernetes.io/projected/d1842ed7-0d43-4f3d-815b-a2f45026a569-kube-api-access-4nxgs\") pod \"migrator-66df5d496b-qx2rc\" (UID: \"d1842ed7-0d43-4f3d-815b-a2f45026a569\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-qx2rc" Nov 02 06:17:04.872954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.872937 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc4fn\" (UniqueName: \"kubernetes.io/projected/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-kube-api-access-dc4fn\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:17:04.873115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.873101 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zljkr\" (UniqueName: \"kubernetes.io/projected/ec8433d1-0c6c-4732-b8a4-d73c251cd5ef-kube-api-access-zljkr\") pod \"network-check-target-k7c7h\" (UID: \"ec8433d1-0c6c-4732-b8a4-d73c251cd5ef\") " pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:17:04.873237 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.873218 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42bxk\" (UniqueName: \"kubernetes.io/projected/0122975d-2172-4e2a-9af8-47f972bc5ecc-kube-api-access-42bxk\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.873276 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.873253 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxbtw\" (UniqueName: \"kubernetes.io/projected/15fd4393-b2d4-4f12-8ae8-f38b92a3b05e-kube-api-access-hxbtw\") pod \"ingress-canary-x9xwq\" (UID: \"15fd4393-b2d4-4f12-8ae8-f38b92a3b05e\") " pod="openshift-ingress-canary/ingress-canary-x9xwq" Nov 02 06:17:04.875428 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.875408 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97d22\" (UniqueName: \"kubernetes.io/projected/720c25e0-1117-4a48-af60-9022daed412a-kube-api-access-97d22\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:04.879144 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.879128 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 02 06:17:04.879783 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.879765 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0122975d-2172-4e2a-9af8-47f972bc5ecc-metrics-client-ca\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.880920 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.880905 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-kubelet\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.880972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.880927 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-bin\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.880972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.880943 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-dbus\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:17:04.880972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.880958 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-device-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.880973 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-var-lib-kubelet\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.880989 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-multus\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881005 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-conf-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881010 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-kubelet\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881013 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-bin\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881029 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881048 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-device-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881062 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-socket-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881066 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-conf-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881064 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-dbus\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881052 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-var-lib-kubelet\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881086 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-multus\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881111 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-etc-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881123 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-kubelet-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881157 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-socket-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881164 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-modprobe-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881173 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-etc-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881192 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-kubernetes\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881251 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-modprobe-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881256 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-lib-modules\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881264 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-kubernetes\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881320 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-slash\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881346 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-slash\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881350 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-lib-modules\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881346 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08d074a5-dadb-4e71-a67f-9bc5c33d8857-host-slash\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881368 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/08d074a5-dadb-4e71-a67f-9bc5c33d8857-host-slash\") pod \"iptables-alerter-8tczj\" (UID: \"08d074a5-dadb-4e71-a67f-9bc5c33d8857\") " pod="openshift-network-operator/iptables-alerter-8tczj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881372 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-ovn\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881389 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-ovn\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881396 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-crio-socket\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881414 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881433 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-run\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881472 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ba1e4bef-666b-4088-8e2b-9f7a55ab1327-crio-socket\") pod \"insights-runtime-extractor-x48kz\" (UID: \"ba1e4bef-666b-4088-8e2b-9f7a55ab1327\") " pod="openshift-insights/insights-runtime-extractor-x48kz" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881478 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-run\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881492 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881527 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1edfb6b-cbff-47e6-aab2-a942731747b8-hosts-file\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881554 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-netns\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881560 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e1edfb6b-cbff-47e6-aab2-a942731747b8-hosts-file\") pod \"node-resolver-5jr5c\" (UID: \"e1edfb6b-cbff-47e6-aab2-a942731747b8\") " pod="openshift-dns/node-resolver-5jr5c" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881579 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-netns\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881584 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-systemd-units\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881607 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-etc-kubernetes\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881629 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-systemd-units\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881632 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-sys\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881661 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-etc-kubernetes\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881678 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-root\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881680 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-sys\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881694 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-netd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881716 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-cnibin\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.881701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881720 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-root\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881732 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-system-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881747 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-systemd\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881756 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-cni-netd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881763 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-system-cni-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881787 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-bin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881793 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-cnibin\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881810 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-systemd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881832 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-system-cni-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881840 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-cni-bin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881837 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-conf\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881815 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-system-cni-dir\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881850 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-systemd\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881872 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-systemd\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881967 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cnibin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881979 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-conf\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.881994 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-registration-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882021 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882029 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-cnibin\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882046 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-host\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882069 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-node-log\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882123 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-node-log\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882035 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-registration-dir\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882134 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-host\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882144 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysctl-d\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882149 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-netns\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882170 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-netns\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882180 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882196 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882212 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882215 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-hostroot\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882240 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882244 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-hostroot\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882255 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysconfig\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882245 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882290 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-sys-fs\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882295 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-etc-selinux\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882324 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-log-socket\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882338 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3100b5d8-88b0-45c8-8e06-d758f2b769fa-sys-fs\") pod \"aws-ebs-csi-driver-node-kjhzc\" (UID: \"3100b5d8-88b0-45c8-8e06-d758f2b769fa\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-kjhzc" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882338 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/513b68de-5cb3-4038-9592-5eb8989af256-etc-sysconfig\") pod \"tuned-ccb2x\" (UID: \"513b68de-5cb3-4038-9592-5eb8989af256\") " pod="openshift-cluster-node-tuning-operator/tuned-ccb2x" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882349 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882364 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-log-socket\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882375 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-sys\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882388 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aed3a242-64df-41a6-98b3-e2a6543918fb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gm9nl\" (UID: \"aed3a242-64df-41a6-98b3-e2a6543918fb\") " pod="openshift-multus/multus-additional-cni-plugins-gm9nl" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882409 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-multus-certs\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882431 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-wtmp\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882448 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-var-lib-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882449 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-sys\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882461 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-multus-certs\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882471 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-kubelet-config\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882487 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0122975d-2172-4e2a-9af8-47f972bc5ecc-node-exporter-wtmp\") pod \"node-exporter-7wr46\" (UID: \"0122975d-2172-4e2a-9af8-47f972bc5ecc\") " pod="openshift-monitoring/node-exporter-7wr46" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882496 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70e61026-bb17-490c-aed0-c7387729c9ef-host\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882497 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-var-lib-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882528 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/70e61026-bb17-490c-aed0-c7387729c9ef-host\") pod \"node-ca-58ckp\" (UID: \"70e61026-bb17-490c-aed0-c7387729c9ef\") " pod="openshift-image-registry/node-ca-58ckp" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882533 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6ab6077a-6df9-4de9-8632-98830b9a8653-kubelet-config\") pod \"global-pull-secret-syncer-4vv5l\" (UID: \"6ab6077a-6df9-4de9-8632-98830b9a8653\") " pod="kube-system/global-pull-secret-syncer-4vv5l" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882561 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882520 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f129d6cd-acad-478c-a2a7-cfa37d4c39f9-run-openvswitch\") pod \"ovnkube-node-x4qlj\" (UID: \"f129d6cd-acad-478c-a2a7-cfa37d4c39f9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882584 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-socket-dir-parent\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882605 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-kubelet\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882631 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-k8s-cni-cncf-io\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882633 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-multus-socket-dir-parent\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882676 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-run-k8s-cni-cncf-io\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.882773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.882688 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11477d6f-1bd9-40c9-8d57-c0a0e1edaebf-host-var-lib-kubelet\") pod \"multus-c9z9s\" (UID: \"11477d6f-1bd9-40c9-8d57-c0a0e1edaebf\") " pod="openshift-multus/multus-c9z9s" Nov 02 06:17:04.885421 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.885405 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 02 06:17:04.886612 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.886589 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/563834d8-a7ee-490b-a123-7358961378a2-trusted-ca\") pod \"image-registry-5b4b56fff9-77j92\" (UID: \"563834d8-a7ee-490b-a123-7358961378a2\") " pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:04.888712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:04.888697 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/720c25e0-1117-4a48-af60-9022daed412a-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-p7227\" (UID: \"720c25e0-1117-4a48-af60-9022daed412a\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" Nov 02 06:17:05.027765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:05.027744 5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:17:05.114739 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:05.114711 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:05.317132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:05.317095 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:06.728919 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:06.728886 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-p7227" event={"ID":"720c25e0-1117-4a48-af60-9022daed412a","Type":"ContainerStarted","Data":"bfebcfba349fd69cd2fb4c56c3a66b06872dea29ae25146e3cabe02093d5fd29"} Nov 02 06:17:07.018270 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:07.018244 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fc9xt" Nov 02 06:17:07.034782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:07.034751 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:07.211248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:07.211216 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:07.398248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:07.398166 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:07.505936 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:07.505903 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wvjp6" Nov 02 06:17:07.733490 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:07.733457 5025 generic.go:356] "Generic (PLEG): container finished" podID="0122975d-2172-4e2a-9af8-47f972bc5ecc" containerID="b27efe53bd36d313e99328574ba288185989083d29bafca0900f714f4c7db858" exitCode=0 Nov 02 06:17:07.733845 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:07.733532 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wr46" event={"ID":"0122975d-2172-4e2a-9af8-47f972bc5ecc","Type":"ContainerDied","Data":"b27efe53bd36d313e99328574ba288185989083d29bafca0900f714f4c7db858"} Nov 02 06:17:07.828863 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:07.828839 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:07.870522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:07.870497 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:07.870706 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:07.870693 5025 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 02 06:17:08.018868 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:08.018794 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:08.738915 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:08.738880 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wr46" event={"ID":"0122975d-2172-4e2a-9af8-47f972bc5ecc","Type":"ContainerStarted","Data":"3071b9739053442ce75d1d9954368b88a3dbd12d98f981e7c8996f388f0467b2"} Nov 02 06:17:08.739285 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:08.738938 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7wr46" event={"ID":"0122975d-2172-4e2a-9af8-47f972bc5ecc","Type":"ContainerStarted","Data":"42b4c99374701fcc8053ad859fdea5dc7785a98f4fc287b66ea0514f6a242dc8"} Nov 02 06:17:09.581744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:09.581711 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b4b56fff9-77j92" Nov 02 06:17:09.691407 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:09.691375 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:09.872297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:09.872221 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:10.064284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:10.064250 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:10.243739 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:10.243692 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:10.418790 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:10.418760 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:11.693234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:11.693200 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:11.936922 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:11.936888 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:12.826291 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:12.826264 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k7c7h" Nov 02 06:17:13.066439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:13.066406 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x4qlj" Nov 02 06:17:13.517918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:13.517884 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:13.739591 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:13.739557 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:14.413711 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:14.413678 5025 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-34.us-west-2.compute.internal" event="NodeReady" Nov 02 06:17:17.462475 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:17.462428 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:17.652476 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:17.652441 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:20.265588 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:20.265521 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:20.446483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:20.446437 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:21.148199 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:21.148156 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:21.325312 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:21.325273 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:21.688035 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:21.688002 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:21.860032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:21.859978 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:22.864461 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:22.864391 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:23.030309 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:23.030251 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:25.465133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:25.465096 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:25.629040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:25.629000 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:32.706562 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:17:32.706524 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:17:36.693963 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:36.693927 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:36.873194 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:36.873160 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:38.189342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:38.189294 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:38.363108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:38.363057 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:41.982834 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:41.982799 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:42.161206 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:42.161174 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:43.992457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:43.992421 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:44.161015 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:44.160979 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:44.333405 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:44.333341 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:47.257192 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.257158 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:47.457590 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.457553 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:17:47.666542 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.666513 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 06:17:47.668959 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.668942 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760271 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760236 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c6b7817-19d9-45c8-a23b-f4f44e513890-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760271 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760280 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-config-volume\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760463 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760300 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7c6b7817-19d9-45c8-a23b-f4f44e513890-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760463 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760381 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760463 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760417 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760456 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6b7817-19d9-45c8-a23b-f4f44e513890-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760596 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760556 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760596 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760580 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c6b7817-19d9-45c8-a23b-f4f44e513890-config-out\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760665 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760598 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n46tr\" (UniqueName: \"kubernetes.io/projected/7c6b7817-19d9-45c8-a23b-f4f44e513890-kube-api-access-n46tr\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760665 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760618 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-web-config\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760725 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760704 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760750 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.760802 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.760776 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c6b7817-19d9-45c8-a23b-f4f44e513890-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.767294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.767271 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Nov 02 06:17:47.767294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.767271 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Nov 02 06:17:47.814487 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.814462 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Nov 02 06:17:47.814697 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.814505 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Nov 02 06:17:47.814697 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.814470 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Nov 02 06:17:47.814697 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.814481 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Nov 02 06:17:47.814697 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.814505 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Nov 02 06:17:47.814697 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.814682 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Nov 02 06:17:47.815107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.815041 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tx8jf\"" Nov 02 06:17:47.861316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861292 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861329 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c6b7817-19d9-45c8-a23b-f4f44e513890-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861378 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c6b7817-19d9-45c8-a23b-f4f44e513890-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861409 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-config-volume\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861433 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7c6b7817-19d9-45c8-a23b-f4f44e513890-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861633 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861536 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861633 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861588 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861736 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861617 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6b7817-19d9-45c8-a23b-f4f44e513890-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861736 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861669 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861736 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861707 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c6b7817-19d9-45c8-a23b-f4f44e513890-config-out\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861887 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861737 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n46tr\" (UniqueName: \"kubernetes.io/projected/7c6b7817-19d9-45c8-a23b-f4f44e513890-kube-api-access-n46tr\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861887 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861740 5025 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 02 06:17:47.861887 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861768 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-web-config\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.861887 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.861815 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.862401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.862376 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7c6b7817-19d9-45c8-a23b-f4f44e513890-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.862690 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.862666 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7c6b7817-19d9-45c8-a23b-f4f44e513890-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.862869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.862846 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Nov 02 06:17:47.863876 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.863845 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7c6b7817-19d9-45c8-a23b-f4f44e513890-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.864799 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.864772 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c6b7817-19d9-45c8-a23b-f4f44e513890-config-out\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.864892 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.864838 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 06:17:47.865008 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.864984 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c6b7817-19d9-45c8-a23b-f4f44e513890-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.865008 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.865001 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.865166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.865055 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.865222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.865159 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.866131 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.866114 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.866584 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.866563 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-config-volume\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.866802 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.866780 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-web-config\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.866993 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.866976 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7c6b7817-19d9-45c8-a23b-f4f44e513890-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.917200 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.917131 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n46tr\" (UniqueName: \"kubernetes.io/projected/7c6b7817-19d9-45c8-a23b-f4f44e513890-kube-api-access-n46tr\") pod \"alertmanager-main-0\" (UID: \"7c6b7817-19d9-45c8-a23b-f4f44e513890\") " pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:47.985249 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:47.985215 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Nov 02 06:17:48.445391 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:48.445357 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Nov 02 06:17:48.493724 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:17:48.493685 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6b7817_19d9_45c8_a23b_f4f44e513890.slice/crio-72814ca4c3f7f72e4906e85d469375b6a344c4bd9fe6de590fd289c600c4369e WatchSource:0}: Error finding container 72814ca4c3f7f72e4906e85d469375b6a344c4bd9fe6de590fd289c600c4369e: Status 404 returned error can't find the container with id 72814ca4c3f7f72e4906e85d469375b6a344c4bd9fe6de590fd289c600c4369e Nov 02 06:17:48.866980 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:48.866903 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c6b7817-19d9-45c8-a23b-f4f44e513890","Type":"ContainerStarted","Data":"72814ca4c3f7f72e4906e85d469375b6a344c4bd9fe6de590fd289c600c4369e"} Nov 02 06:17:50.879437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:50.879400 5025 generic.go:356] "Generic (PLEG): container finished" podID="7c6b7817-19d9-45c8-a23b-f4f44e513890" containerID="d6667e231499d9498c346e384618425b3552249d72255a33d832ece770468195" exitCode=0 Nov 02 06:17:50.879824 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:50.879459 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c6b7817-19d9-45c8-a23b-f4f44e513890","Type":"ContainerDied","Data":"d6667e231499d9498c346e384618425b3552249d72255a33d832ece770468195"} Nov 02 06:17:53.891811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:53.891734 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c6b7817-19d9-45c8-a23b-f4f44e513890","Type":"ContainerStarted","Data":"e4fea16b032fec44b9db4979dedbbdc481c98a93b5c9b981c06e955f6216f3a7"} Nov 02 06:17:53.891811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:53.891796 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c6b7817-19d9-45c8-a23b-f4f44e513890","Type":"ContainerStarted","Data":"66406535bcbd38dba5418a641a80ae532bcc27e6c6ca1d405e691124b34933db"} Nov 02 06:17:53.892235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:53.891821 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c6b7817-19d9-45c8-a23b-f4f44e513890","Type":"ContainerStarted","Data":"2a3655c0e2601290336f8a73a63cbe292db8055a4e25d7dca6f994324d159072"} Nov 02 06:17:53.892235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:53.891844 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c6b7817-19d9-45c8-a23b-f4f44e513890","Type":"ContainerStarted","Data":"75a21add85708d0c61bbd5a8c944968827c0022d2746b0e73d8e598b514442fd"} Nov 02 06:17:53.892235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:53.891870 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c6b7817-19d9-45c8-a23b-f4f44e513890","Type":"ContainerStarted","Data":"e2b95df0e9e3243a1f4007aeb469e63cc37d579db4068c7e58a85454dff8c544"} Nov 02 06:17:54.901238 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:54.901204 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7c6b7817-19d9-45c8-a23b-f4f44e513890","Type":"ContainerStarted","Data":"adc4d9356a59db7abbddf9358a0af6587437d749d7adef0e9cd6757e8d62e225"} Nov 02 06:17:55.161296 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:17:55.161187 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.849760543 podStartE2EDuration="8.161172549s" totalImagesPullingTime="5.299633796s" totalInitContainerRuntime="11.77821ms" isStatefulPod=false podCreationTimestamp="2026-11-02 06:17:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:17:55.113584756 +0000 UTC m=+51.561961516" watchObservedRunningTime="2026-11-02 06:17:55.161172549 +0000 UTC m=+51.609549302" Nov 02 06:18:02.706549 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:18:02.706509 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:18:13.814374 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:18:13.814319 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:18:32.705688 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:18:32.705644 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:18:57.770167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:18:57.770061 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:19:02.706699 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:19:02.706660 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:19:14.137131 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:19:14.137097 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:19:32.705624 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:19:32.705586 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:19:46.397002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:19:46.396965 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:19:56.930673 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:19:56.930639 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:19:57.113136 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:19:57.113104 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:00.913595 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:00.913564 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:01.112370 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:01.112339 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:01.321188 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:01.321115 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:01.974808 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:01.974778 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l"] Nov 02 06:20:01.978058 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:01.978040 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:02.078205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.078166 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-tt4bx\"" Nov 02 06:20:02.078205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.078184 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqkbw\" (UniqueName: \"kubernetes.io/projected/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-kube-api-access-xqkbw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:02.078205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.078185 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 02 06:20:02.078467 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.078204 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 02 06:20:02.078467 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.078299 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:02.078467 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.078329 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:02.078882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.078862 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l"] Nov 02 06:20:02.179370 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.179327 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:02.179370 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.179371 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:02.179570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.179413 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqkbw\" (UniqueName: \"kubernetes.io/projected/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-kube-api-access-xqkbw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:02.179708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.179688 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:02.179782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.179764 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:02.235806 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.235746 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqkbw\" (UniqueName: \"kubernetes.io/projected/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-kube-api-access-xqkbw\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:02.295527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.295499 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:02.396420 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.396389 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82"] Nov 02 06:20:02.399583 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.399566 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.453441 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.453409 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vk64l\"" Nov 02 06:20:02.455739 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.455720 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 02 06:20:02.455836 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.455785 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 02 06:20:02.455960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.455946 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 02 06:20:02.457063 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.457049 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 02 06:20:02.457176 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.457066 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 02 06:20:02.457176 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.457099 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 02 06:20:02.463580 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.463556 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82"] Nov 02 06:20:02.467538 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.467514 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:02.583513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.583423 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.583513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.583470 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.583691 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.583583 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f300abe-e19b-4c52-88d3-7949276f0054-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.583691 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.583612 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.583691 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.583636 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.583809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.583723 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.583809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.583742 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssl78\" (UniqueName: \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-kube-api-access-ssl78\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.662253 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.662196 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l"] Nov 02 06:20:02.664750 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:20:02.664728 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b89f4f7_ca57_4288_876b_64ccdcb9edeb.slice/crio-9fadc04c13b21d0438957e2d1899433ff81b497896507573a11941a37239e854 WatchSource:0}: Error finding container 9fadc04c13b21d0438957e2d1899433ff81b497896507573a11941a37239e854: Status 404 returned error can't find the container with id 9fadc04c13b21d0438957e2d1899433ff81b497896507573a11941a37239e854 Nov 02 06:20:02.684966 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.684941 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f300abe-e19b-4c52-88d3-7949276f0054-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.685119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.684974 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.685119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.685001 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.685119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.685044 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.685119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.685069 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssl78\" (UniqueName: \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-kube-api-access-ssl78\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.685329 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.685154 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.685329 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.685193 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.685674 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.685642 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.687214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.687187 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f300abe-e19b-4c52-88d3-7949276f0054-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.687318 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.687301 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.687431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.687412 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.687570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.687553 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.706953 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:20:02.706921 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:20:02.707189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.707167 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:02.740138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.740118 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:02.740347 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:02.740328 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssl78\" (UniqueName: \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-kube-api-access-ssl78\") pod \"istiod-openshift-gateway-7876bfcb74-6jv82\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:03.011144 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:03.011098 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:03.301540 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:03.301461 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" event={"ID":"3b89f4f7-ca57-4288-876b-64ccdcb9edeb","Type":"ContainerStarted","Data":"9fadc04c13b21d0438957e2d1899433ff81b497896507573a11941a37239e854"} Nov 02 06:20:03.430923 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:03.430787 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82"] Nov 02 06:20:03.432971 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:20:03.432939 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f300abe_e19b_4c52_88d3_7949276f0054.slice/crio-9eaccb4d1d8f21e10345b8e962b73914a303eee106f1994ce939ee7a177a0a88 WatchSource:0}: Error finding container 9eaccb4d1d8f21e10345b8e962b73914a303eee106f1994ce939ee7a177a0a88: Status 404 returned error can't find the container with id 9eaccb4d1d8f21e10345b8e962b73914a303eee106f1994ce939ee7a177a0a88 Nov 02 06:20:03.570431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:03.570311 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:04.306315 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:04.306281 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" event={"ID":"2f300abe-e19b-4c52-88d3-7949276f0054","Type":"ContainerStarted","Data":"9eaccb4d1d8f21e10345b8e962b73914a303eee106f1994ce939ee7a177a0a88"} Nov 02 06:20:11.207855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:11.207821 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:20:11.208238 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:11.207895 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:20:11.334176 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:11.334125 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" event={"ID":"2f300abe-e19b-4c52-88d3-7949276f0054","Type":"ContainerStarted","Data":"7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6"} Nov 02 06:20:11.334328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:11.334222 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:11.474005 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:11.473886 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" podStartSLOduration=1.701912231 podStartE2EDuration="9.473866608s" totalImagesPullingTime="7.771954377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:20:11.473249043 +0000 UTC m=+187.921625819" watchObservedRunningTime="2026-11-02 06:20:11.473866608 +0000 UTC m=+187.922243364" Nov 02 06:20:12.340836 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:12.340808 5025 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-6jv82 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:20:12.341222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:12.340858 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" podUID="2f300abe-e19b-4c52-88d3-7949276f0054" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:20:12.727105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:12.727059 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:12.899132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:12.899100 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:13.267892 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:13.267858 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:13.344095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:13.344046 5025 generic.go:356] "Generic (PLEG): container finished" podID="3b89f4f7-ca57-4288-876b-64ccdcb9edeb" containerID="e5f12f3fa2a792e012ab9d0eecc82a6b11775455e61652ca20330721894226f6" exitCode=0 Nov 02 06:20:13.344458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:13.344138 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" event={"ID":"3b89f4f7-ca57-4288-876b-64ccdcb9edeb","Type":"ContainerDied","Data":"e5f12f3fa2a792e012ab9d0eecc82a6b11775455e61652ca20330721894226f6"} Nov 02 06:20:13.541387 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:13.541308 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:13.727605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:13.727575 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:14.477542 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:14.477510 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:15.339776 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:15.339744 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:20:17.286530 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:17.286500 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:18.360877 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:18.360842 5025 generic.go:356] "Generic (PLEG): container finished" podID="3b89f4f7-ca57-4288-876b-64ccdcb9edeb" containerID="ac2750d01dc60af2de6660230c5ae0fcce250221c5aafa3d8e854282ae1c9153" exitCode=0 Nov 02 06:20:18.361310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:18.360934 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" event={"ID":"3b89f4f7-ca57-4288-876b-64ccdcb9edeb","Type":"ContainerDied","Data":"ac2750d01dc60af2de6660230c5ae0fcce250221c5aafa3d8e854282ae1c9153"} Nov 02 06:20:28.395604 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:28.395565 5025 generic.go:356] "Generic (PLEG): container finished" podID="3b89f4f7-ca57-4288-876b-64ccdcb9edeb" containerID="07580da10b06c78be49949c444f6dbedd9905f41a1451f048594adf5ec9d1ffc" exitCode=0 Nov 02 06:20:28.396016 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:28.395609 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" event={"ID":"3b89f4f7-ca57-4288-876b-64ccdcb9edeb","Type":"ContainerDied","Data":"07580da10b06c78be49949c444f6dbedd9905f41a1451f048594adf5ec9d1ffc"} Nov 02 06:20:29.592973 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:29.592945 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:29.704655 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:29.704627 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-kube-api-access-xqkbw\" (UniqueName: \"kubernetes.io/projected/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-kube-api-access-xqkbw\") pod \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " Nov 02 06:20:29.704806 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:29.704683 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-bundle\") pod \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " Nov 02 06:20:29.704806 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:29.704707 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-util\" (UniqueName: \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-util\") pod \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\" (UID: \"3b89f4f7-ca57-4288-876b-64ccdcb9edeb\") " Nov 02 06:20:29.705092 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:29.705053 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-bundle" pod "3b89f4f7-ca57-4288-876b-64ccdcb9edeb" (UID: "3b89f4f7-ca57-4288-876b-64ccdcb9edeb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:29.706823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:29.706794 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-kube-api-access-xqkbw" pod "3b89f4f7-ca57-4288-876b-64ccdcb9edeb" (UID: "3b89f4f7-ca57-4288-876b-64ccdcb9edeb"). InnerVolumeSpecName "kube-api-access-xqkbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:20:29.709951 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:29.709923 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-util" pod "3b89f4f7-ca57-4288-876b-64ccdcb9edeb" (UID: "3b89f4f7-ca57-4288-876b-64ccdcb9edeb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:20:29.806085 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:29.806056 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqkbw\" (UniqueName: \"kubernetes.io/projected/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-kube-api-access-xqkbw\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:29.806221 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:29.806097 5025 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-bundle\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:29.806221 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:29.806107 5025 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3b89f4f7-ca57-4288-876b-64ccdcb9edeb-util\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:20:30.404302 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:30.404264 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" event={"ID":"3b89f4f7-ca57-4288-876b-64ccdcb9edeb","Type":"ContainerDied","Data":"9fadc04c13b21d0438957e2d1899433ff81b497896507573a11941a37239e854"} Nov 02 06:20:30.404469 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:30.404313 5025 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fadc04c13b21d0438957e2d1899433ff81b497896507573a11941a37239e854" Nov 02 06:20:30.404469 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:30.404311 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64479g9l" Nov 02 06:20:32.705936 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:20:32.705902 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:20:58.082599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:58.082527 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:58.270656 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:58.270624 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:59.232496 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:59.232451 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:59.454744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:59.454710 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:20:59.944541 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:20:59.944448 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:00.143332 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.143300 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68"] Nov 02 06:21:00.143551 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.143540 5025 cpu_manager.go:414] "containerMap: removing container" podUID="3b89f4f7-ca57-4288-876b-64ccdcb9edeb" containerName="pull" Nov 02 06:21:00.143593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.143553 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b89f4f7-ca57-4288-876b-64ccdcb9edeb" containerName="pull" Nov 02 06:21:00.143593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.143562 5025 cpu_manager.go:414] "containerMap: removing container" podUID="3b89f4f7-ca57-4288-876b-64ccdcb9edeb" containerName="extract" Nov 02 06:21:00.143593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.143567 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b89f4f7-ca57-4288-876b-64ccdcb9edeb" containerName="extract" Nov 02 06:21:00.143681 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.143604 5025 cpu_manager.go:414] "containerMap: removing container" podUID="3b89f4f7-ca57-4288-876b-64ccdcb9edeb" containerName="util" Nov 02 06:21:00.143681 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.143609 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b89f4f7-ca57-4288-876b-64ccdcb9edeb" containerName="util" Nov 02 06:21:00.143681 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.143654 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b89f4f7-ca57-4288-876b-64ccdcb9edeb" containerName="extract" Nov 02 06:21:00.162531 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.162504 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.261513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.261325 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 02 06:21:00.261513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.261362 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 02 06:21:00.261513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.261337 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 02 06:21:00.261513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.261437 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 02 06:21:00.261513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.261448 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5hv8f\"" Nov 02 06:21:00.262466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.261787 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 02 06:21:00.262688 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.262667 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68"] Nov 02 06:21:00.315854 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.315812 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:00.320936 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.320914 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82dbbd34-1f42-46d7-bbba-03ae3659ff45-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.321032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.320946 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82dbbd34-1f42-46d7-bbba-03ae3659ff45-metrics-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.321032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.320966 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82dbbd34-1f42-46d7-bbba-03ae3659ff45-manifests\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.321169 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.321146 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82dbbd34-1f42-46d7-bbba-03ae3659ff45-tmp\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.321205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.321190 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82dbbd34-1f42-46d7-bbba-03ae3659ff45-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.321275 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.321257 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvth7\" (UniqueName: \"kubernetes.io/projected/82dbbd34-1f42-46d7-bbba-03ae3659ff45-kube-api-access-nvth7\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.422345 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.422316 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82dbbd34-1f42-46d7-bbba-03ae3659ff45-tmp\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.422497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.422353 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82dbbd34-1f42-46d7-bbba-03ae3659ff45-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.422497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.422479 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvth7\" (UniqueName: \"kubernetes.io/projected/82dbbd34-1f42-46d7-bbba-03ae3659ff45-kube-api-access-nvth7\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.422574 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.422514 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82dbbd34-1f42-46d7-bbba-03ae3659ff45-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.422574 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.422545 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82dbbd34-1f42-46d7-bbba-03ae3659ff45-metrics-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.422712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.422688 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82dbbd34-1f42-46d7-bbba-03ae3659ff45-manifests\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.423036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.423017 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/82dbbd34-1f42-46d7-bbba-03ae3659ff45-manifests\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.424692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.424659 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/82dbbd34-1f42-46d7-bbba-03ae3659ff45-tmp\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.425005 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.424986 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/82dbbd34-1f42-46d7-bbba-03ae3659ff45-webhook-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.425071 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.425033 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/82dbbd34-1f42-46d7-bbba-03ae3659ff45-apiservice-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.425160 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.425133 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/82dbbd34-1f42-46d7-bbba-03ae3659ff45-metrics-cert\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.478034 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.478008 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvth7\" (UniqueName: \"kubernetes.io/projected/82dbbd34-1f42-46d7-bbba-03ae3659ff45-kube-api-access-nvth7\") pod \"opendatahub-operator-controller-manager-565d9484c7-ccr68\" (UID: \"82dbbd34-1f42-46d7-bbba-03ae3659ff45\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.494829 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.494800 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:00.503623 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:00.503595 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:01.022965 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:01.022934 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68"] Nov 02 06:21:01.071383 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:21:01.071353 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82dbbd34_1f42_46d7_bbba_03ae3659ff45.slice/crio-4f14a66100a0fd8b54bf3df1e7a902841521f8bf2c952f05921db58db75446ec WatchSource:0}: Error finding container 4f14a66100a0fd8b54bf3df1e7a902841521f8bf2c952f05921db58db75446ec: Status 404 returned error can't find the container with id 4f14a66100a0fd8b54bf3df1e7a902841521f8bf2c952f05921db58db75446ec Nov 02 06:21:01.508550 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:01.508497 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" event={"ID":"82dbbd34-1f42-46d7-bbba-03ae3659ff45","Type":"ContainerStarted","Data":"4f14a66100a0fd8b54bf3df1e7a902841521f8bf2c952f05921db58db75446ec"} Nov 02 06:21:02.712960 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:21:02.712927 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:21:03.229239 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:03.229207 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:03.462664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:03.462626 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:04.418896 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:04.418834 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:05.615787 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:05.615726 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:06.210772 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:06.210706 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:06.534363 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:06.534326 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" event={"ID":"82dbbd34-1f42-46d7-bbba-03ae3659ff45","Type":"ContainerStarted","Data":"6e3212cde625a20f04b168f3f2fb7f29c1b28aaa38ecef6b054b8867513a6bf0"} Nov 02 06:21:07.059318 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:07.059287 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:07.249145 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:07.249049 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:07.539115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:07.539006 5025 generic.go:356] "Generic (PLEG): container finished" podID="82dbbd34-1f42-46d7-bbba-03ae3659ff45" containerID="6e3212cde625a20f04b168f3f2fb7f29c1b28aaa38ecef6b054b8867513a6bf0" exitCode=0 Nov 02 06:21:07.539264 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:07.539112 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" event={"ID":"82dbbd34-1f42-46d7-bbba-03ae3659ff45","Type":"ContainerDied","Data":"6e3212cde625a20f04b168f3f2fb7f29c1b28aaa38ecef6b054b8867513a6bf0"} Nov 02 06:21:08.544382 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:08.544347 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" event={"ID":"82dbbd34-1f42-46d7-bbba-03ae3659ff45","Type":"ContainerStarted","Data":"518c923d71cb0936b09b20983acac97b0e28fceb00ac7356d05783980af92e88"} Nov 02 06:21:08.544828 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:08.544508 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:08.762068 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:08.762016 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" podStartSLOduration=3.047699312 podStartE2EDuration="9.762003075s" totalImagesPullingTime="6.232658026s" totalInitContainerRuntime="481.645737ms" isStatefulPod=false podCreationTimestamp="2026-11-02 06:20:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:08.714298183 +0000 UTC m=+245.162674983" watchObservedRunningTime="2026-11-02 06:21:08.762003075 +0000 UTC m=+245.210379829" Nov 02 06:21:13.023948 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:13.023905 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:13.675413 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:13.675376 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:14.610414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:14.610382 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:14.838995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:14.838964 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:15.477697 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:15.477657 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:17.624816 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:17.624785 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:18.133232 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:18.133202 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:19.550647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:19.550619 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-565d9484c7-ccr68" Nov 02 06:21:20.638815 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:20.638779 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:20.852816 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:20.852782 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:21.035856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:21.035820 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:21.316305 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:21.316213 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:21.527497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:21.527456 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:23.455127 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:23.455071 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:24.885996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:24.885964 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:25.270861 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:25.270834 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:25.487227 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:25.487179 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:27.432599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:27.432565 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:28.113680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:28.113635 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:28.616665 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:28.616637 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:28.805575 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:28.805540 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:29.031462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:29.031431 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:29.218905 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:29.218862 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:29.398747 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:29.398667 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:29.580949 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:29.580912 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:30.230326 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:30.230278 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:30.474436 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:30.474402 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:32.706444 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:21:32.706411 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:21:33.571721 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:33.571683 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:33.757790 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:33.757742 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:34.012871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:34.012838 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:35.369421 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.369381 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:35.390744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.390705 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hbsbz"] Nov 02 06:21:35.394068 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.394044 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:35.447641 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.447607 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hbsbz"] Nov 02 06:21:35.479047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.479008 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ad4d0a47-9396-4eb9-bf14-0c982d272de4-config-file\") pod \"limitador-limitador-74dfcfb894-hbsbz\" (UID: \"ad4d0a47-9396-4eb9-bf14-0c982d272de4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:35.479243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.479067 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z45w\" (UniqueName: \"kubernetes.io/projected/ad4d0a47-9396-4eb9-bf14-0c982d272de4-kube-api-access-2z45w\") pod \"limitador-limitador-74dfcfb894-hbsbz\" (UID: \"ad4d0a47-9396-4eb9-bf14-0c982d272de4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:35.493604 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.493566 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 06:21:35.493604 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.493570 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 02 06:21:35.493811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.493608 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 06:21:35.493811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.493779 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mj4ct\"" Nov 02 06:21:35.563204 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.563170 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-784594468b-pffjp"] Nov 02 06:21:35.566233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.566218 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" Nov 02 06:21:35.579792 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.579767 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cdac39f-8bc9-4943-98e8-173af92400bf-tls-certs\") pod \"kube-auth-proxy-784594468b-pffjp\" (UID: \"0cdac39f-8bc9-4943-98e8-173af92400bf\") " pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" Nov 02 06:21:35.579917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.579813 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cdac39f-8bc9-4943-98e8-173af92400bf-tmp\") pod \"kube-auth-proxy-784594468b-pffjp\" (UID: \"0cdac39f-8bc9-4943-98e8-173af92400bf\") " pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" Nov 02 06:21:35.579917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.579887 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ad4d0a47-9396-4eb9-bf14-0c982d272de4-config-file\") pod \"limitador-limitador-74dfcfb894-hbsbz\" (UID: \"ad4d0a47-9396-4eb9-bf14-0c982d272de4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:35.579917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.579915 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4lc6\" (UniqueName: \"kubernetes.io/projected/0cdac39f-8bc9-4943-98e8-173af92400bf-kube-api-access-n4lc6\") pod \"kube-auth-proxy-784594468b-pffjp\" (UID: \"0cdac39f-8bc9-4943-98e8-173af92400bf\") " pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" Nov 02 06:21:35.580058 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.579956 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z45w\" (UniqueName: \"kubernetes.io/projected/ad4d0a47-9396-4eb9-bf14-0c982d272de4-kube-api-access-2z45w\") pod \"limitador-limitador-74dfcfb894-hbsbz\" (UID: \"ad4d0a47-9396-4eb9-bf14-0c982d272de4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:35.580627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.580607 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ad4d0a47-9396-4eb9-bf14-0c982d272de4-config-file\") pod \"limitador-limitador-74dfcfb894-hbsbz\" (UID: \"ad4d0a47-9396-4eb9-bf14-0c982d272de4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:35.593299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.593274 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:35.619342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.619309 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-784594468b-pffjp"] Nov 02 06:21:35.620183 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.620122 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 02 06:21:35.620328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.620175 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-g6qjm\"" Nov 02 06:21:35.620328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.620265 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 02 06:21:35.634647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.634607 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z45w\" (UniqueName: \"kubernetes.io/projected/ad4d0a47-9396-4eb9-bf14-0c982d272de4-kube-api-access-2z45w\") pod \"limitador-limitador-74dfcfb894-hbsbz\" (UID: \"ad4d0a47-9396-4eb9-bf14-0c982d272de4\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:35.641988 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.641960 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hbsbz"] Nov 02 06:21:35.681056 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.681027 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cdac39f-8bc9-4943-98e8-173af92400bf-tls-certs\") pod \"kube-auth-proxy-784594468b-pffjp\" (UID: \"0cdac39f-8bc9-4943-98e8-173af92400bf\") " pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" Nov 02 06:21:35.681241 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.681071 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cdac39f-8bc9-4943-98e8-173af92400bf-tmp\") pod \"kube-auth-proxy-784594468b-pffjp\" (UID: \"0cdac39f-8bc9-4943-98e8-173af92400bf\") " pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" Nov 02 06:21:35.681241 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.681113 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4lc6\" (UniqueName: \"kubernetes.io/projected/0cdac39f-8bc9-4943-98e8-173af92400bf-kube-api-access-n4lc6\") pod \"kube-auth-proxy-784594468b-pffjp\" (UID: \"0cdac39f-8bc9-4943-98e8-173af92400bf\") " pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" Nov 02 06:21:35.683394 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.683364 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cdac39f-8bc9-4943-98e8-173af92400bf-tmp\") pod \"kube-auth-proxy-784594468b-pffjp\" (UID: \"0cdac39f-8bc9-4943-98e8-173af92400bf\") " pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" Nov 02 06:21:35.683568 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.683549 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0cdac39f-8bc9-4943-98e8-173af92400bf-tls-certs\") pod \"kube-auth-proxy-784594468b-pffjp\" (UID: \"0cdac39f-8bc9-4943-98e8-173af92400bf\") " pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" Nov 02 06:21:35.706447 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.706411 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:35.737983 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.737948 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4lc6\" (UniqueName: \"kubernetes.io/projected/0cdac39f-8bc9-4943-98e8-173af92400bf-kube-api-access-n4lc6\") pod \"kube-auth-proxy-784594468b-pffjp\" (UID: \"0cdac39f-8bc9-4943-98e8-173af92400bf\") " pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" Nov 02 06:21:35.799227 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.799196 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:35.877965 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:35.877875 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" Nov 02 06:21:36.066925 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:36.066898 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hbsbz"] Nov 02 06:21:36.069249 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:21:36.069223 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4d0a47_9396_4eb9_bf14_0c982d272de4.slice/crio-58eacb8fecd5effc1597132c0e2ced267dae84e031c3c5d85c21fa394162b359 WatchSource:0}: Error finding container 58eacb8fecd5effc1597132c0e2ced267dae84e031c3c5d85c21fa394162b359: Status 404 returned error can't find the container with id 58eacb8fecd5effc1597132c0e2ced267dae84e031c3c5d85c21fa394162b359 Nov 02 06:21:36.070683 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:36.070655 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:36.237855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:36.237825 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-784594468b-pffjp"] Nov 02 06:21:36.240720 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:21:36.240690 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdac39f_8bc9_4943_98e8_173af92400bf.slice/crio-042ec1e73d920c9f36f7fa78ee92ae26f278230148697ba1bf3caf6c486f468f WatchSource:0}: Error finding container 042ec1e73d920c9f36f7fa78ee92ae26f278230148697ba1bf3caf6c486f468f: Status 404 returned error can't find the container with id 042ec1e73d920c9f36f7fa78ee92ae26f278230148697ba1bf3caf6c486f468f Nov 02 06:21:36.268878 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:36.268845 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:36.459531 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:36.459492 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:36.642619 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:36.642540 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" event={"ID":"0cdac39f-8bc9-4943-98e8-173af92400bf","Type":"ContainerStarted","Data":"042ec1e73d920c9f36f7fa78ee92ae26f278230148697ba1bf3caf6c486f468f"} Nov 02 06:21:36.643833 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:36.643811 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" event={"ID":"ad4d0a47-9396-4eb9-bf14-0c982d272de4","Type":"ContainerStarted","Data":"58eacb8fecd5effc1597132c0e2ced267dae84e031c3c5d85c21fa394162b359"} Nov 02 06:21:38.003547 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:38.003489 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:38.250308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:38.250222 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:38.452385 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:38.452341 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:40.663009 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:40.662978 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" event={"ID":"0cdac39f-8bc9-4943-98e8-173af92400bf","Type":"ContainerStarted","Data":"810a92e1252d8b918537efa545501c15ab5c423fe88c0b92cb2a36d62b1455e2"} Nov 02 06:21:40.664295 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:40.664274 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" event={"ID":"ad4d0a47-9396-4eb9-bf14-0c982d272de4","Type":"ContainerStarted","Data":"826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6"} Nov 02 06:21:40.664436 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:40.664416 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:40.772546 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:40.772469 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-784594468b-pffjp" podStartSLOduration=2.035456828 podStartE2EDuration="5.772457095s" totalImagesPullingTime="3.737000267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:40.771867099 +0000 UTC m=+277.220243873" watchObservedRunningTime="2026-11-02 06:21:40.772457095 +0000 UTC m=+277.220833877" Nov 02 06:21:40.883560 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:40.883492 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" podStartSLOduration=1.7910779319999999 podStartE2EDuration="5.883477646s" totalImagesPullingTime="4.092399714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:21:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:21:40.882505309 +0000 UTC m=+277.330882087" watchObservedRunningTime="2026-11-02 06:21:40.883477646 +0000 UTC m=+277.331854462" Nov 02 06:21:41.423311 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:41.423283 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:41.632364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:41.632334 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:42.876828 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:42.876785 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:44.319824 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:44.319735 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:47.437571 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:47.437526 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:47.621877 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:47.621843 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:51.668665 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:51.668637 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:52.436642 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:52.436609 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:52.614705 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:52.614675 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:52.799718 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:52.799635 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:52.987221 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:52.987189 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:53.234454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:53.234419 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:53.419741 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:53.419707 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:55.216800 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:55.216765 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:55.559698 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:55.559609 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:55.750666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:55.750633 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:57.081530 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:57.081497 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:57.262763 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:57.262727 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:21:57.507032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:57.507004 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hbsbz"] Nov 02 06:21:57.507300 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:57.507277 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" podUID="ad4d0a47-9396-4eb9-bf14-0c982d272de4" containerName="limitador" containerID="cri-o://826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6" gracePeriod=30 Nov 02 06:21:58.044988 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.044959 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:58.159480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.159402 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4d0a47-9396-4eb9-bf14-0c982d272de4-kube-api-access-2z45w\" (UniqueName: \"kubernetes.io/projected/ad4d0a47-9396-4eb9-bf14-0c982d272de4-kube-api-access-2z45w\") pod \"ad4d0a47-9396-4eb9-bf14-0c982d272de4\" (UID: \"ad4d0a47-9396-4eb9-bf14-0c982d272de4\") " Nov 02 06:21:58.159900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.159493 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4d0a47-9396-4eb9-bf14-0c982d272de4-config-file\" (UniqueName: \"kubernetes.io/configmap/ad4d0a47-9396-4eb9-bf14-0c982d272de4-config-file\") pod \"ad4d0a47-9396-4eb9-bf14-0c982d272de4\" (UID: \"ad4d0a47-9396-4eb9-bf14-0c982d272de4\") " Nov 02 06:21:58.159941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.159885 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4d0a47-9396-4eb9-bf14-0c982d272de4-config-file" pod "ad4d0a47-9396-4eb9-bf14-0c982d272de4" (UID: "ad4d0a47-9396-4eb9-bf14-0c982d272de4"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:21:58.161530 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.161509 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4d0a47-9396-4eb9-bf14-0c982d272de4-kube-api-access-2z45w" pod "ad4d0a47-9396-4eb9-bf14-0c982d272de4" (UID: "ad4d0a47-9396-4eb9-bf14-0c982d272de4"). InnerVolumeSpecName "kube-api-access-2z45w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:21:58.260178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.260143 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z45w\" (UniqueName: \"kubernetes.io/projected/ad4d0a47-9396-4eb9-bf14-0c982d272de4-kube-api-access-2z45w\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:21:58.260178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.260171 5025 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ad4d0a47-9396-4eb9-bf14-0c982d272de4-config-file\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:21:58.726765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.726731 5025 generic.go:356] "Generic (PLEG): container finished" podID="ad4d0a47-9396-4eb9-bf14-0c982d272de4" containerID="826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6" exitCode=0 Nov 02 06:21:58.726953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.726771 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" Nov 02 06:21:58.726953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.726822 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" event={"ID":"ad4d0a47-9396-4eb9-bf14-0c982d272de4","Type":"ContainerDied","Data":"826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6"} Nov 02 06:21:58.726953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.726873 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-hbsbz" event={"ID":"ad4d0a47-9396-4eb9-bf14-0c982d272de4","Type":"ContainerDied","Data":"58eacb8fecd5effc1597132c0e2ced267dae84e031c3c5d85c21fa394162b359"} Nov 02 06:21:58.726953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.726895 5025 scope.go:122] "RemoveContainer" containerID="826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6" Nov 02 06:21:58.740381 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.740363 5025 scope.go:122] "RemoveContainer" containerID="826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6" Nov 02 06:21:58.740683 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:21:58.740661 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6\": container with ID starting with 826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6 not found: ID does not exist" containerID="826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6" Nov 02 06:21:58.740747 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.740689 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6"} err="failed to get container status \"826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6\": rpc error: code = NotFound desc = could not find container \"826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6\": container with ID starting with 826e66521d9c12c93b5298a906062f91da891e25fad80e06b947d450c46f9ab6 not found: ID does not exist" Nov 02 06:21:58.881524 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.881481 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hbsbz"] Nov 02 06:21:58.928835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:58.928803 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-hbsbz"] Nov 02 06:21:59.666175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:21:59.666147 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4d0a47-9396-4eb9-bf14-0c982d272de4" path="/var/lib/kubelet/pods/ad4d0a47-9396-4eb9-bf14-0c982d272de4/volumes" Nov 02 06:22:00.241446 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:00.241411 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:00.419493 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:00.419461 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:02.695995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:02.695956 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:02.707663 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:22:02.707623 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:22:02.875545 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:02.875471 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:08.461548 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.461509 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lt9ll"] Nov 02 06:22:08.462040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.461799 5025 cpu_manager.go:414] "containerMap: removing container" podUID="ad4d0a47-9396-4eb9-bf14-0c982d272de4" containerName="limitador" Nov 02 06:22:08.462040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.461811 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4d0a47-9396-4eb9-bf14-0c982d272de4" containerName="limitador" Nov 02 06:22:08.462040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.461869 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4d0a47-9396-4eb9-bf14-0c982d272de4" containerName="limitador" Nov 02 06:22:08.465944 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.465927 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" Nov 02 06:22:08.516423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.516394 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lt9ll"] Nov 02 06:22:08.537163 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.537133 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl5vj\" (UniqueName: \"kubernetes.io/projected/a6645bd1-1eb2-461a-9f64-094109cd2190-kube-api-access-tl5vj\") pod \"authorino-56c6c9ddd5-lt9ll\" (UID: \"a6645bd1-1eb2-461a-9f64-094109cd2190\") " pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" Nov 02 06:22:08.562829 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.562806 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 02 06:22:08.562829 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.562815 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-x84wk\"" Nov 02 06:22:08.562829 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.562824 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 02 06:22:08.638241 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.638196 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl5vj\" (UniqueName: \"kubernetes.io/projected/a6645bd1-1eb2-461a-9f64-094109cd2190-kube-api-access-tl5vj\") pod \"authorino-56c6c9ddd5-lt9ll\" (UID: \"a6645bd1-1eb2-461a-9f64-094109cd2190\") " pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" Nov 02 06:22:08.692497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.692469 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl5vj\" (UniqueName: \"kubernetes.io/projected/a6645bd1-1eb2-461a-9f64-094109cd2190-kube-api-access-tl5vj\") pod \"authorino-56c6c9ddd5-lt9ll\" (UID: \"a6645bd1-1eb2-461a-9f64-094109cd2190\") " pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" Nov 02 06:22:08.787920 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:08.787847 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" Nov 02 06:22:09.147053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:09.146984 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lt9ll"] Nov 02 06:22:09.149608 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:22:09.149580 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6645bd1_1eb2_461a_9f64_094109cd2190.slice/crio-85004229dfce6ad1e6ee9c42703251b359f67b65b26b375fb4580db8b599096d WatchSource:0}: Error finding container 85004229dfce6ad1e6ee9c42703251b359f67b65b26b375fb4580db8b599096d: Status 404 returned error can't find the container with id 85004229dfce6ad1e6ee9c42703251b359f67b65b26b375fb4580db8b599096d Nov 02 06:22:09.151062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:09.151045 5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:22:09.310266 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:09.310234 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:09.769155 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:09.769116 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" event={"ID":"a6645bd1-1eb2-461a-9f64-094109cd2190","Type":"ContainerStarted","Data":"85004229dfce6ad1e6ee9c42703251b359f67b65b26b375fb4580db8b599096d"} Nov 02 06:22:11.332255 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:11.332212 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:12.028342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.028306 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z"] Nov 02 06:22:12.031060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.031043 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" Nov 02 06:22:12.084398 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.084372 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z"] Nov 02 06:22:12.130423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.130392 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 02 06:22:12.130423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.130405 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-pcsdt\"" Nov 02 06:22:12.130423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.130405 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 02 06:22:12.130659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.130468 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 02 06:22:12.168899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.168839 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnd9\" (UniqueName: \"kubernetes.io/projected/a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f-kube-api-access-7dnd9\") pod \"postgres-7cdbf6dc44-fxx2z\" (UID: \"a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" Nov 02 06:22:12.169113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.169040 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f-data\") pod \"postgres-7cdbf6dc44-fxx2z\" (UID: \"a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" Nov 02 06:22:12.171333 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.171269 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:12.270023 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.269988 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnd9\" (UniqueName: \"kubernetes.io/projected/a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f-kube-api-access-7dnd9\") pod \"postgres-7cdbf6dc44-fxx2z\" (UID: \"a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" Nov 02 06:22:12.270221 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.270061 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f-data\") pod \"postgres-7cdbf6dc44-fxx2z\" (UID: \"a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" Nov 02 06:22:12.270482 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.270463 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f-data\") pod \"postgres-7cdbf6dc44-fxx2z\" (UID: \"a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" Nov 02 06:22:12.324062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.323998 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnd9\" (UniqueName: \"kubernetes.io/projected/a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f-kube-api-access-7dnd9\") pod \"postgres-7cdbf6dc44-fxx2z\" (UID: \"a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" Nov 02 06:22:12.342067 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.342034 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" Nov 02 06:22:12.703131 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.703100 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z"] Nov 02 06:22:12.705940 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:22:12.705904 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda16c03ab_56a9_4ab9_9cbc_6cfa18a1f25f.slice/crio-ee3e2d5132aaceeb066f90c04e2a1a71c95206b95091a2f9e6fc66627276e8be WatchSource:0}: Error finding container ee3e2d5132aaceeb066f90c04e2a1a71c95206b95091a2f9e6fc66627276e8be: Status 404 returned error can't find the container with id ee3e2d5132aaceeb066f90c04e2a1a71c95206b95091a2f9e6fc66627276e8be Nov 02 06:22:12.782626 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:12.782587 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" event={"ID":"a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f","Type":"ContainerStarted","Data":"ee3e2d5132aaceeb066f90c04e2a1a71c95206b95091a2f9e6fc66627276e8be"} Nov 02 06:22:13.786968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:13.786933 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" event={"ID":"a6645bd1-1eb2-461a-9f64-094109cd2190","Type":"ContainerStarted","Data":"663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571"} Nov 02 06:22:13.899821 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:13.899771 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" podStartSLOduration=2.061998795 podStartE2EDuration="5.899758703s" totalImagesPullingTime="3.837759908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:22:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:22:13.893773681 +0000 UTC m=+310.342150471" watchObservedRunningTime="2026-11-02 06:22:13.899758703 +0000 UTC m=+310.348135462" Nov 02 06:22:15.177448 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:15.177416 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:21.819405 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:21.819366 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" event={"ID":"a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f","Type":"ContainerStarted","Data":"2f3ad3beb3a765c61d3fe40c28d9bd553bcbd1c559d28269082b4fc8f199ac8e"} Nov 02 06:22:21.819801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:21.819517 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" Nov 02 06:22:21.927867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:21.927807 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" podStartSLOduration=2.003349235 podStartE2EDuration="10.927788826s" totalImagesPullingTime="8.924439591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:22:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:22:21.927506535 +0000 UTC m=+318.375883311" watchObservedRunningTime="2026-11-02 06:22:21.927788826 +0000 UTC m=+318.376165582" Nov 02 06:22:27.852896 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:27.852863 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-fxx2z" Nov 02 06:22:32.706701 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:22:32.706664 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:22:34.747485 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:34.747446 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:34.938787 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:34.938754 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:35.115712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:35.115627 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:39.509994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:39.509937 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:39.705554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:39.705510 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:42.953530 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:42.953502 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:43.141789 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:43.141757 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:45.547960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.547925 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9"] Nov 02 06:22:45.551259 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.551239 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.650567 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.650534 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9"] Nov 02 06:22:45.738573 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.738533 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c2505e-f91e-4a17-a828-eab6162b785a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.738573 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.738574 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.738766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.738602 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.738766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.738726 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5c2505e-f91e-4a17-a828-eab6162b785a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.738766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.738758 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.738874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.738855 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.738908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.738895 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shvfg\" (UniqueName: \"kubernetes.io/projected/d5c2505e-f91e-4a17-a828-eab6162b785a-kube-api-access-shvfg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.840209 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.840124 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c2505e-f91e-4a17-a828-eab6162b785a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.840209 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.840167 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.840209 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.840198 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.840422 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.840394 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5c2505e-f91e-4a17-a828-eab6162b785a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.840465 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.840432 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.840541 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.840522 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.840596 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.840553 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shvfg\" (UniqueName: \"kubernetes.io/projected/d5c2505e-f91e-4a17-a828-eab6162b785a-kube-api-access-shvfg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.841118 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.841057 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.842522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.842495 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5c2505e-f91e-4a17-a828-eab6162b785a-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.842695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.842678 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.842737 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.842709 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d5c2505e-f91e-4a17-a828-eab6162b785a-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.842818 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.842796 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.896300 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.896260 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5c2505e-f91e-4a17-a828-eab6162b785a-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:45.896467 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:45.896340 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shvfg\" (UniqueName: \"kubernetes.io/projected/d5c2505e-f91e-4a17-a828-eab6162b785a-kube-api-access-shvfg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-twkd9\" (UID: \"d5c2505e-f91e-4a17-a828-eab6162b785a\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:46.162325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:46.162301 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:46.569954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:46.569928 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9"] Nov 02 06:22:46.572040 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:22:46.572013 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c2505e_f91e_4a17_a828_eab6162b785a.slice/crio-26a4a5d22bf6bef22251dc934ee658b1c3ccfd420ed20d7100016970677b48bd WatchSource:0}: Error finding container 26a4a5d22bf6bef22251dc934ee658b1c3ccfd420ed20d7100016970677b48bd: Status 404 returned error can't find the container with id 26a4a5d22bf6bef22251dc934ee658b1c3ccfd420ed20d7100016970677b48bd Nov 02 06:22:46.573986 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:46.573954 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:22:46.574064 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:46.574019 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:22:46.905348 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:46.905267 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" event={"ID":"d5c2505e-f91e-4a17-a828-eab6162b785a","Type":"ContainerStarted","Data":"c0bc02e758fca9da34be6fb06e589e3ffd674439a90bb76a7449064988836da0"} Nov 02 06:22:46.905348 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:46.905328 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" event={"ID":"d5c2505e-f91e-4a17-a828-eab6162b785a","Type":"ContainerStarted","Data":"26a4a5d22bf6bef22251dc934ee658b1c3ccfd420ed20d7100016970677b48bd"} Nov 02 06:22:46.905591 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:46.905432 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:47.068047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:47.067991 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" podStartSLOduration=2.067976691 podStartE2EDuration="2.067976691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:22:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:22:47.02219724 +0000 UTC m=+343.470574008" watchObservedRunningTime="2026-11-02 06:22:47.067976691 +0000 UTC m=+343.516353446" Nov 02 06:22:47.910868 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:47.910819 5025 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-twkd9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:22:47.911258 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:47.910879 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" podUID="d5c2505e-f91e-4a17-a828-eab6162b785a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:22:50.310987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:50.310953 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:50.488004 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:50.487965 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:50.667018 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:50.666987 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:50.843276 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:50.843245 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:50.910062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:50.910033 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-twkd9" Nov 02 06:22:51.028014 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.027933 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:51.103392 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.103345 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82"] Nov 02 06:22:51.103609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.103586 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" podUID="2f300abe-e19b-4c52-88d3-7949276f0054" containerName="discovery" containerID="cri-o://7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6" gracePeriod=30 Nov 02 06:22:51.222718 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.222687 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:51.345241 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.345213 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:22:51.381365 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.381328 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-dns-cert\") pod \"2f300abe-e19b-4c52-88d3-7949276f0054\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " Nov 02 06:22:51.381365 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.381370 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-kubeconfig\") pod \"2f300abe-e19b-4c52-88d3-7949276f0054\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " Nov 02 06:22:51.381600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.381452 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f300abe-e19b-4c52-88d3-7949276f0054-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f300abe-e19b-4c52-88d3-7949276f0054-local-certs\") pod \"2f300abe-e19b-4c52-88d3-7949276f0054\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " Nov 02 06:22:51.381600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.381480 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-kube-api-access-ssl78\" (UniqueName: \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-kube-api-access-ssl78\") pod \"2f300abe-e19b-4c52-88d3-7949276f0054\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " Nov 02 06:22:51.381600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.381532 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-ca-configmap\") pod \"2f300abe-e19b-4c52-88d3-7949276f0054\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " Nov 02 06:22:51.381600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.381568 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-istio-token\" (UniqueName: \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-istio-token\") pod \"2f300abe-e19b-4c52-88d3-7949276f0054\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " Nov 02 06:22:51.381808 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.381696 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-cacerts\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-cacerts\") pod \"2f300abe-e19b-4c52-88d3-7949276f0054\" (UID: \"2f300abe-e19b-4c52-88d3-7949276f0054\") " Nov 02 06:22:51.381992 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.381954 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-ca-configmap" pod "2f300abe-e19b-4c52-88d3-7949276f0054" (UID: "2f300abe-e19b-4c52-88d3-7949276f0054"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:22:51.383845 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.383813 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f300abe-e19b-4c52-88d3-7949276f0054-local-certs" pod "2f300abe-e19b-4c52-88d3-7949276f0054" (UID: "2f300abe-e19b-4c52-88d3-7949276f0054"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:22:51.383964 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.383940 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-dns-cert" pod "2f300abe-e19b-4c52-88d3-7949276f0054" (UID: "2f300abe-e19b-4c52-88d3-7949276f0054"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:22:51.383964 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.383948 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-istio-token" pod "2f300abe-e19b-4c52-88d3-7949276f0054" (UID: "2f300abe-e19b-4c52-88d3-7949276f0054"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:22:51.384113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.384016 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-kube-api-access-ssl78" pod "2f300abe-e19b-4c52-88d3-7949276f0054" (UID: "2f300abe-e19b-4c52-88d3-7949276f0054"). InnerVolumeSpecName "kube-api-access-ssl78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:22:51.384175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.384148 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-cacerts" pod "2f300abe-e19b-4c52-88d3-7949276f0054" (UID: "2f300abe-e19b-4c52-88d3-7949276f0054"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:22:51.384245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.384229 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-kubeconfig" pod "2f300abe-e19b-4c52-88d3-7949276f0054" (UID: "2f300abe-e19b-4c52-88d3-7949276f0054"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:22:51.482712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.482676 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssl78\" (UniqueName: \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-kube-api-access-ssl78\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:22:51.482712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.482707 5025 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-ca-configmap\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:22:51.482712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.482717 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f300abe-e19b-4c52-88d3-7949276f0054-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:22:51.482923 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.482726 5025 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-cacerts\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:22:51.482923 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.482735 5025 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-csr-dns-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:22:51.482923 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.482743 5025 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2f300abe-e19b-4c52-88d3-7949276f0054-istio-kubeconfig\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:22:51.482923 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.482752 5025 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f300abe-e19b-4c52-88d3-7949276f0054-local-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:22:51.924452 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.924360 5025 generic.go:356] "Generic (PLEG): container finished" podID="2f300abe-e19b-4c52-88d3-7949276f0054" containerID="7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6" exitCode=0 Nov 02 06:22:51.924452 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.924412 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" Nov 02 06:22:51.924645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.924456 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" event={"ID":"2f300abe-e19b-4c52-88d3-7949276f0054","Type":"ContainerDied","Data":"7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6"} Nov 02 06:22:51.924645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.924512 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" event={"ID":"2f300abe-e19b-4c52-88d3-7949276f0054","Type":"ContainerDied","Data":"9eaccb4d1d8f21e10345b8e962b73914a303eee106f1994ce939ee7a177a0a88"} Nov 02 06:22:51.924645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.924538 5025 scope.go:122] "RemoveContainer" containerID="7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6" Nov 02 06:22:51.932630 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.932610 5025 scope.go:122] "RemoveContainer" containerID="7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6" Nov 02 06:22:51.932883 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:22:51.932866 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6\": container with ID starting with 7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6 not found: ID does not exist" containerID="7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6" Nov 02 06:22:51.932939 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:51.932896 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6"} err="failed to get container status \"7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6\": rpc error: code = NotFound desc = could not find container \"7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6\": container with ID starting with 7a6c4e6bdd6bf6fd541db3c5c4ed4358a22d6d71e6612f44695da999586aefc6 not found: ID does not exist" Nov 02 06:22:52.079549 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:52.079516 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82"] Nov 02 06:22:52.126675 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:52.126643 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82"] Nov 02 06:22:52.384339 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:52.384261 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:53.671132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:53.671099 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f300abe-e19b-4c52-88d3-7949276f0054" path="/var/lib/kubelet/pods/2f300abe-e19b-4c52-88d3-7949276f0054/volumes" Nov 02 06:22:56.339647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:56.339601 5025 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-6jv82 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.14:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:22:56.340031 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:56.339669 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-6jv82" podUID="2f300abe-e19b-4c52-88d3-7949276f0054" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.14:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:22:57.181116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:57.181061 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:57.374428 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:57.374395 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:57.553427 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:57.553341 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:57.748376 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:57.748344 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:57.951901 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:57.951865 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:58.132758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:58.132724 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:22:58.320711 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:22:58.320642 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:02.706580 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:23:02.706542 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:23:12.799647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:12.799611 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:12.982270 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:12.982236 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:14.942043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:14.941963 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:15.136313 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:15.136279 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:15.320284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:15.320212 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:15.501926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:15.501890 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:15.998037 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:15.997997 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc"] Nov 02 06:23:15.998434 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:15.998314 5025 cpu_manager.go:414] "containerMap: removing container" podUID="2f300abe-e19b-4c52-88d3-7949276f0054" containerName="discovery" Nov 02 06:23:15.998434 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:15.998325 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f300abe-e19b-4c52-88d3-7949276f0054" containerName="discovery" Nov 02 06:23:15.998434 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:15.998378 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f300abe-e19b-4c52-88d3-7949276f0054" containerName="discovery" Nov 02 06:23:16.001043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.001021 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.070130 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.070095 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxpdh\" (UniqueName: \"kubernetes.io/projected/c5ef166b-a092-4f19-8814-f09295e3d9f9-kube-api-access-nxpdh\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.070295 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.070165 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-maas-api-tls\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.070295 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.070199 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5ef166b-a092-4f19-8814-f09295e3d9f9-openshift-service-ca\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.070295 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.070234 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-metrics-certs\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.097644 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.097610 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gm8x6\"" Nov 02 06:23:16.097803 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.097732 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 02 06:23:16.097803 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.097733 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 02 06:23:16.098915 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.098894 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc"] Nov 02 06:23:16.171419 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.171378 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-metrics-certs\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.171619 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.171443 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxpdh\" (UniqueName: \"kubernetes.io/projected/c5ef166b-a092-4f19-8814-f09295e3d9f9-kube-api-access-nxpdh\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.171619 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.171467 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-maas-api-tls\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.171619 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.171497 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5ef166b-a092-4f19-8814-f09295e3d9f9-openshift-service-ca\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.172215 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.172194 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5ef166b-a092-4f19-8814-f09295e3d9f9-openshift-service-ca\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.173898 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.173872 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-metrics-certs\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.173988 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.173921 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-maas-api-tls\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.228465 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.228428 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxpdh\" (UniqueName: \"kubernetes.io/projected/c5ef166b-a092-4f19-8814-f09295e3d9f9-kube-api-access-nxpdh\") pod \"maas-api-6d669c7fcd-w2lmc\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.314166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.314060 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:16.673261 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.673234 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc"] Nov 02 06:23:16.675871 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:23:16.675838 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5ef166b_a092_4f19_8814_f09295e3d9f9.slice/crio-3f93b488ebe536758e67329c5ccd06772f7bd6234ff0ce589899d2ce07c1e3ca WatchSource:0}: Error finding container 3f93b488ebe536758e67329c5ccd06772f7bd6234ff0ce589899d2ce07c1e3ca: Status 404 returned error can't find the container with id 3f93b488ebe536758e67329c5ccd06772f7bd6234ff0ce589899d2ce07c1e3ca Nov 02 06:23:16.745349 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.745299 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:16.925834 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:16.925748 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:17.016358 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:17.016322 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" event={"ID":"c5ef166b-a092-4f19-8814-f09295e3d9f9","Type":"ContainerStarted","Data":"3f93b488ebe536758e67329c5ccd06772f7bd6234ff0ce589899d2ce07c1e3ca"} Nov 02 06:23:17.109765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:17.109721 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:21.033500 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:21.033464 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" event={"ID":"c5ef166b-a092-4f19-8814-f09295e3d9f9","Type":"ContainerStarted","Data":"cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb"} Nov 02 06:23:21.033895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:21.033687 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:21.143666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:21.143600 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" podStartSLOduration=2.789498594 podStartE2EDuration="6.14358427s" totalImagesPullingTime="3.354085676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:23:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:23:21.143286263 +0000 UTC m=+377.591663038" watchObservedRunningTime="2026-11-02 06:23:21.14358427 +0000 UTC m=+377.591961021" Nov 02 06:23:27.042907 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:27.042870 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:27.601936 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:27.601901 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:27.827226 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:27.827191 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jlrnh"] Nov 02 06:23:27.830454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:27.830438 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" Nov 02 06:23:27.853198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:27.853128 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:23:27.883839 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:27.883808 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jlrnh"] Nov 02 06:23:27.972930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:27.972890 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vfpk\" (UniqueName: \"kubernetes.io/projected/f33a8fef-cddb-4e2b-9630-6b3d08fa5b36-kube-api-access-7vfpk\") pod \"authorino-6f6944bcdd-jlrnh\" (UID: \"f33a8fef-cddb-4e2b-9630-6b3d08fa5b36\") " pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" Nov 02 06:23:28.008039 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.008008 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj"] Nov 02 06:23:28.011250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.011230 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.065047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.065013 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj"] Nov 02 06:23:28.074286 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.074259 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h29j4\" (UniqueName: \"kubernetes.io/projected/ed97b347-05b9-4efe-b1d4-2427cf047d03-kube-api-access-h29j4\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.074475 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.074298 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-maas-api-tls\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.074475 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.074326 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed97b347-05b9-4efe-b1d4-2427cf047d03-openshift-service-ca\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.074475 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.074436 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vfpk\" (UniqueName: \"kubernetes.io/projected/f33a8fef-cddb-4e2b-9630-6b3d08fa5b36-kube-api-access-7vfpk\") pod \"authorino-6f6944bcdd-jlrnh\" (UID: \"f33a8fef-cddb-4e2b-9630-6b3d08fa5b36\") " pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" Nov 02 06:23:28.074626 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.074482 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-metrics-certs\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.128744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.128675 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vfpk\" (UniqueName: \"kubernetes.io/projected/f33a8fef-cddb-4e2b-9630-6b3d08fa5b36-kube-api-access-7vfpk\") pod \"authorino-6f6944bcdd-jlrnh\" (UID: \"f33a8fef-cddb-4e2b-9630-6b3d08fa5b36\") " pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" Nov 02 06:23:28.143437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.143402 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" Nov 02 06:23:28.175402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.175365 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-metrics-certs\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.175586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.175434 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h29j4\" (UniqueName: \"kubernetes.io/projected/ed97b347-05b9-4efe-b1d4-2427cf047d03-kube-api-access-h29j4\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.175586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.175452 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-maas-api-tls\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.175586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.175469 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed97b347-05b9-4efe-b1d4-2427cf047d03-openshift-service-ca\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.176126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.176099 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed97b347-05b9-4efe-b1d4-2427cf047d03-openshift-service-ca\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.177993 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.177969 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-metrics-certs\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.178208 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.178113 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-maas-api-tls\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.182066 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.180000 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jlrnh"] Nov 02 06:23:28.229457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.229430 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h29j4\" (UniqueName: \"kubernetes.io/projected/ed97b347-05b9-4efe-b1d4-2427cf047d03-kube-api-access-h29j4\") pod \"maas-api-7dc858b877-qxvmj\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.322544 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.322500 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:28.501731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.501706 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jlrnh"] Nov 02 06:23:28.504007 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:23:28.503981 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf33a8fef_cddb_4e2b_9630_6b3d08fa5b36.slice/crio-9af480b1be02a1d5c72a1ad88c3ee738bd9bb93bc782718f05b80dd998612073 WatchSource:0}: Error finding container 9af480b1be02a1d5c72a1ad88c3ee738bd9bb93bc782718f05b80dd998612073: Status 404 returned error can't find the container with id 9af480b1be02a1d5c72a1ad88c3ee738bd9bb93bc782718f05b80dd998612073 Nov 02 06:23:28.686324 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:28.686301 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj"] Nov 02 06:23:28.688660 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:23:28.688620 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded97b347_05b9_4efe_b1d4_2427cf047d03.slice/crio-4a5ef395d9f63b5743cd0948c9137f3d20a645cc3ec38c9889451d44e674a493 WatchSource:0}: Error finding container 4a5ef395d9f63b5743cd0948c9137f3d20a645cc3ec38c9889451d44e674a493: Status 404 returned error can't find the container with id 4a5ef395d9f63b5743cd0948c9137f3d20a645cc3ec38c9889451d44e674a493 Nov 02 06:23:29.062621 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:29.062582 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" event={"ID":"ed97b347-05b9-4efe-b1d4-2427cf047d03","Type":"ContainerStarted","Data":"4a5ef395d9f63b5743cd0948c9137f3d20a645cc3ec38c9889451d44e674a493"} Nov 02 06:23:29.063806 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:29.063778 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" event={"ID":"f33a8fef-cddb-4e2b-9630-6b3d08fa5b36","Type":"ContainerStarted","Data":"9af480b1be02a1d5c72a1ad88c3ee738bd9bb93bc782718f05b80dd998612073"} Nov 02 06:23:30.074048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:30.073955 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" event={"ID":"ed97b347-05b9-4efe-b1d4-2427cf047d03","Type":"ContainerStarted","Data":"5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9"} Nov 02 06:23:30.074454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:30.074062 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:30.181733 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:30.181684 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" podStartSLOduration=2.200424464 podStartE2EDuration="3.181669634s" totalImagesPullingTime="981.24517ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:23:30.181303881 +0000 UTC m=+386.629680664" watchObservedRunningTime="2026-11-02 06:23:30.181669634 +0000 UTC m=+386.630046393" Nov 02 06:23:31.083017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:31.082957 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" event={"ID":"f33a8fef-cddb-4e2b-9630-6b3d08fa5b36","Type":"ContainerStarted","Data":"ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32"} Nov 02 06:23:31.083523 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:31.083027 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" podUID="f33a8fef-cddb-4e2b-9630-6b3d08fa5b36" containerName="authorino" containerID="cri-o://ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32" gracePeriod=30 Nov 02 06:23:31.189821 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:31.189767 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" podStartSLOduration=2.147776908 podStartE2EDuration="4.189751051s" totalImagesPullingTime="2.041974143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:23:31.189506239 +0000 UTC m=+387.637882996" watchObservedRunningTime="2026-11-02 06:23:31.189751051 +0000 UTC m=+387.638127806" Nov 02 06:23:31.332545 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:31.332518 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" Nov 02 06:23:31.402727 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:31.402630 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33a8fef-cddb-4e2b-9630-6b3d08fa5b36-kube-api-access-7vfpk\" (UniqueName: \"kubernetes.io/projected/f33a8fef-cddb-4e2b-9630-6b3d08fa5b36-kube-api-access-7vfpk\") pod \"f33a8fef-cddb-4e2b-9630-6b3d08fa5b36\" (UID: \"f33a8fef-cddb-4e2b-9630-6b3d08fa5b36\") " Nov 02 06:23:31.404678 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:31.404646 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33a8fef-cddb-4e2b-9630-6b3d08fa5b36-kube-api-access-7vfpk" pod "f33a8fef-cddb-4e2b-9630-6b3d08fa5b36" (UID: "f33a8fef-cddb-4e2b-9630-6b3d08fa5b36"). InnerVolumeSpecName "kube-api-access-7vfpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:23:31.503373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:31.503340 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vfpk\" (UniqueName: \"kubernetes.io/projected/f33a8fef-cddb-4e2b-9630-6b3d08fa5b36-kube-api-access-7vfpk\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:23:31.622608 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:31.622568 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lt9ll"] Nov 02 06:23:31.622782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:31.622740 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" podUID="a6645bd1-1eb2-461a-9f64-094109cd2190" containerName="authorino" containerID="cri-o://663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571" gracePeriod=30 Nov 02 06:23:31.858760 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:31.858730 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" Nov 02 06:23:32.007630 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.007549 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6645bd1-1eb2-461a-9f64-094109cd2190-kube-api-access-tl5vj\" (UniqueName: \"kubernetes.io/projected/a6645bd1-1eb2-461a-9f64-094109cd2190-kube-api-access-tl5vj\") pod \"a6645bd1-1eb2-461a-9f64-094109cd2190\" (UID: \"a6645bd1-1eb2-461a-9f64-094109cd2190\") " Nov 02 06:23:32.009578 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.009551 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6645bd1-1eb2-461a-9f64-094109cd2190-kube-api-access-tl5vj" pod "a6645bd1-1eb2-461a-9f64-094109cd2190" (UID: "a6645bd1-1eb2-461a-9f64-094109cd2190"). InnerVolumeSpecName "kube-api-access-tl5vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:23:32.088323 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.088290 5025 generic.go:356] "Generic (PLEG): container finished" podID="a6645bd1-1eb2-461a-9f64-094109cd2190" containerID="663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571" exitCode=0 Nov 02 06:23:32.088781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.088334 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" Nov 02 06:23:32.088781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.088378 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" event={"ID":"a6645bd1-1eb2-461a-9f64-094109cd2190","Type":"ContainerDied","Data":"663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571"} Nov 02 06:23:32.088781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.088425 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-lt9ll" event={"ID":"a6645bd1-1eb2-461a-9f64-094109cd2190","Type":"ContainerDied","Data":"85004229dfce6ad1e6ee9c42703251b359f67b65b26b375fb4580db8b599096d"} Nov 02 06:23:32.088781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.088446 5025 scope.go:122] "RemoveContainer" containerID="663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571" Nov 02 06:23:32.089757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.089625 5025 generic.go:356] "Generic (PLEG): container finished" podID="f33a8fef-cddb-4e2b-9630-6b3d08fa5b36" containerID="ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32" exitCode=0 Nov 02 06:23:32.089757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.089670 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" event={"ID":"f33a8fef-cddb-4e2b-9630-6b3d08fa5b36","Type":"ContainerDied","Data":"ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32"} Nov 02 06:23:32.089757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.089701 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" event={"ID":"f33a8fef-cddb-4e2b-9630-6b3d08fa5b36","Type":"ContainerDied","Data":"9af480b1be02a1d5c72a1ad88c3ee738bd9bb93bc782718f05b80dd998612073"} Nov 02 06:23:32.089757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.089673 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-jlrnh" Nov 02 06:23:32.096963 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.096947 5025 scope.go:122] "RemoveContainer" containerID="663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571" Nov 02 06:23:32.097251 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:23:32.097230 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571\": container with ID starting with 663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571 not found: ID does not exist" containerID="663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571" Nov 02 06:23:32.097309 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.097259 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571"} err="failed to get container status \"663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571\": rpc error: code = NotFound desc = could not find container \"663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571\": container with ID starting with 663687e07a0d94110f7896087901ee08ace19aaf5efca12e287a08270cbc7571 not found: ID does not exist" Nov 02 06:23:32.097309 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.097277 5025 scope.go:122] "RemoveContainer" containerID="ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32" Nov 02 06:23:32.104097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.104069 5025 scope.go:122] "RemoveContainer" containerID="ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32" Nov 02 06:23:32.104340 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:23:32.104321 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32\": container with ID starting with ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32 not found: ID does not exist" containerID="ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32" Nov 02 06:23:32.104401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.104350 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32"} err="failed to get container status \"ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32\": rpc error: code = NotFound desc = could not find container \"ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32\": container with ID starting with ae788ce846de937ff738d4b1de4e8eeecaa976692529969a7c825756292e5a32 not found: ID does not exist" Nov 02 06:23:32.108712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.108695 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl5vj\" (UniqueName: \"kubernetes.io/projected/a6645bd1-1eb2-461a-9f64-094109cd2190-kube-api-access-tl5vj\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:23:32.242988 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.242950 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jlrnh"] Nov 02 06:23:32.246816 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.246793 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-jlrnh"] Nov 02 06:23:32.350991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.350911 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lt9ll"] Nov 02 06:23:32.356976 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:32.356948 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-lt9ll"] Nov 02 06:23:32.705705 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:23:32.705671 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:23:33.666257 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:33.666229 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6645bd1-1eb2-461a-9f64-094109cd2190" path="/var/lib/kubelet/pods/a6645bd1-1eb2-461a-9f64-094109cd2190/volumes" Nov 02 06:23:33.666601 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:33.666523 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33a8fef-cddb-4e2b-9630-6b3d08fa5b36" path="/var/lib/kubelet/pods/f33a8fef-cddb-4e2b-9630-6b3d08fa5b36/volumes" Nov 02 06:23:36.088911 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.088876 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 06:23:36.245688 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.245659 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc"] Nov 02 06:23:36.245893 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.245867 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" podUID="c5ef166b-a092-4f19-8814-f09295e3d9f9" containerName="maas-api" containerID="cri-o://cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb" gracePeriod=30 Nov 02 06:23:36.480274 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.480250 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:36.647507 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.647419 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5ef166b-a092-4f19-8814-f09295e3d9f9-kube-api-access-nxpdh\" (UniqueName: \"kubernetes.io/projected/c5ef166b-a092-4f19-8814-f09295e3d9f9-kube-api-access-nxpdh\") pod \"c5ef166b-a092-4f19-8814-f09295e3d9f9\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " Nov 02 06:23:36.647507 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.647459 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-metrics-certs\") pod \"c5ef166b-a092-4f19-8814-f09295e3d9f9\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " Nov 02 06:23:36.647507 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.647500 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5ef166b-a092-4f19-8814-f09295e3d9f9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5ef166b-a092-4f19-8814-f09295e3d9f9-openshift-service-ca\") pod \"c5ef166b-a092-4f19-8814-f09295e3d9f9\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " Nov 02 06:23:36.647722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.647544 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-maas-api-tls\") pod \"c5ef166b-a092-4f19-8814-f09295e3d9f9\" (UID: \"c5ef166b-a092-4f19-8814-f09295e3d9f9\") " Nov 02 06:23:36.647890 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.647868 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5ef166b-a092-4f19-8814-f09295e3d9f9-openshift-service-ca" pod "c5ef166b-a092-4f19-8814-f09295e3d9f9" (UID: "c5ef166b-a092-4f19-8814-f09295e3d9f9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:23:36.649794 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.649756 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-maas-api-tls" pod "c5ef166b-a092-4f19-8814-f09295e3d9f9" (UID: "c5ef166b-a092-4f19-8814-f09295e3d9f9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:23:36.649903 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.649817 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-metrics-certs" pod "c5ef166b-a092-4f19-8814-f09295e3d9f9" (UID: "c5ef166b-a092-4f19-8814-f09295e3d9f9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:23:36.649903 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.649826 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5ef166b-a092-4f19-8814-f09295e3d9f9-kube-api-access-nxpdh" pod "c5ef166b-a092-4f19-8814-f09295e3d9f9" (UID: "c5ef166b-a092-4f19-8814-f09295e3d9f9"). InnerVolumeSpecName "kube-api-access-nxpdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:23:36.748175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.748140 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:23:36.748175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.748171 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxpdh\" (UniqueName: \"kubernetes.io/projected/c5ef166b-a092-4f19-8814-f09295e3d9f9-kube-api-access-nxpdh\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:23:36.748175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.748183 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5ef166b-a092-4f19-8814-f09295e3d9f9-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:23:36.748405 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:36.748192 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5ef166b-a092-4f19-8814-f09295e3d9f9-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:23:37.111745 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:37.111712 5025 generic.go:356] "Generic (PLEG): container finished" podID="c5ef166b-a092-4f19-8814-f09295e3d9f9" containerID="cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb" exitCode=0 Nov 02 06:23:37.112172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:37.111756 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" Nov 02 06:23:37.112172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:37.111780 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" event={"ID":"c5ef166b-a092-4f19-8814-f09295e3d9f9","Type":"ContainerDied","Data":"cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb"} Nov 02 06:23:37.112172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:37.111822 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc" event={"ID":"c5ef166b-a092-4f19-8814-f09295e3d9f9","Type":"ContainerDied","Data":"3f93b488ebe536758e67329c5ccd06772f7bd6234ff0ce589899d2ce07c1e3ca"} Nov 02 06:23:37.112172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:37.111844 5025 scope.go:122] "RemoveContainer" containerID="cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb" Nov 02 06:23:37.120173 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:37.120153 5025 scope.go:122] "RemoveContainer" containerID="cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb" Nov 02 06:23:37.120465 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:23:37.120443 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb\": container with ID starting with cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb not found: ID does not exist" containerID="cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb" Nov 02 06:23:37.120509 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:37.120473 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb"} err="failed to get container status \"cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb\": rpc error: code = NotFound desc = could not find container \"cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb\": container with ID starting with cc601c0cf9e37acbbdc03d1b6bc756c8c35cabec75686fcd70620e70ac20dceb not found: ID does not exist" Nov 02 06:23:37.266458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:37.266425 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc"] Nov 02 06:23:37.313424 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:37.313393 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-w2lmc"] Nov 02 06:23:37.665835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:23:37.665804 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5ef166b-a092-4f19-8814-f09295e3d9f9" path="/var/lib/kubelet/pods/c5ef166b-a092-4f19-8814-f09295e3d9f9/volumes" Nov 02 06:24:02.705377 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:24:02.705338 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:24:15.831971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:24:15.831918 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:24:32.705636 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:24:32.705601 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:24:47.653229 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:24:47.653148 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:24:47.847660 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:24:47.847611 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:02.706206 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:25:02.706165 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:25:07.508643 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.508611 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5"] Nov 02 06:25:07.509158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.508887 5025 cpu_manager.go:414] "containerMap: removing container" podUID="c5ef166b-a092-4f19-8814-f09295e3d9f9" containerName="maas-api" Nov 02 06:25:07.509158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.508898 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5ef166b-a092-4f19-8814-f09295e3d9f9" containerName="maas-api" Nov 02 06:25:07.509158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.508919 5025 cpu_manager.go:414] "containerMap: removing container" podUID="a6645bd1-1eb2-461a-9f64-094109cd2190" containerName="authorino" Nov 02 06:25:07.509158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.508924 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6645bd1-1eb2-461a-9f64-094109cd2190" containerName="authorino" Nov 02 06:25:07.509158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.508930 5025 cpu_manager.go:414] "containerMap: removing container" podUID="f33a8fef-cddb-4e2b-9630-6b3d08fa5b36" containerName="authorino" Nov 02 06:25:07.509158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.508935 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33a8fef-cddb-4e2b-9630-6b3d08fa5b36" containerName="authorino" Nov 02 06:25:07.509158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.508980 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5ef166b-a092-4f19-8814-f09295e3d9f9" containerName="maas-api" Nov 02 06:25:07.509158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.508989 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6645bd1-1eb2-461a-9f64-094109cd2190" containerName="authorino" Nov 02 06:25:07.509158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.508994 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33a8fef-cddb-4e2b-9630-6b3d08fa5b36" containerName="authorino" Nov 02 06:25:07.511839 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.511823 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.562535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.562506 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 02 06:25:07.565791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.565751 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.565791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.565786 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.566000 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.565872 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.566000 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.565922 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.566000 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.565984 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9j2f\" (UniqueName: \"kubernetes.io/projected/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-kube-api-access-d9j2f\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.566175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.566032 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.609310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.609279 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lxzd7\"" Nov 02 06:25:07.609310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.609279 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 02 06:25:07.609520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.609384 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Nov 02 06:25:07.610550 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.610533 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5"] Nov 02 06:25:07.666470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.666443 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.666632 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.666482 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.666632 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.666501 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.666632 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.666525 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.666632 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.666558 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.666632 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.666574 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9j2f\" (UniqueName: \"kubernetes.io/projected/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-kube-api-access-d9j2f\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.666901 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.666875 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.666970 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.666914 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.667024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.666975 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.668766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.668743 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.669088 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.669062 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.722534 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.722502 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9j2f\" (UniqueName: \"kubernetes.io/projected/1fe9a903-1df8-4c6b-9a6f-1c5bb1168116-kube-api-access-d9j2f\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-btbz5\" (UID: \"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:07.765586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.765500 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:07.823770 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:07.823735 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:08.188976 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:08.188952 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5"] Nov 02 06:25:08.190739 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:25:08.190713 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe9a903_1df8_4c6b_9a6f_1c5bb1168116.slice/crio-7cc313e12f1ce9b9ea3fe229bc28513559e2f1b644a00cc2a9faa79d20b93685 WatchSource:0}: Error finding container 7cc313e12f1ce9b9ea3fe229bc28513559e2f1b644a00cc2a9faa79d20b93685: Status 404 returned error can't find the container with id 7cc313e12f1ce9b9ea3fe229bc28513559e2f1b644a00cc2a9faa79d20b93685 Nov 02 06:25:08.438889 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:08.438811 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" event={"ID":"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116","Type":"ContainerStarted","Data":"7cc313e12f1ce9b9ea3fe229bc28513559e2f1b644a00cc2a9faa79d20b93685"} Nov 02 06:25:10.449429 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:10.449399 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" event={"ID":"1fe9a903-1df8-4c6b-9a6f-1c5bb1168116","Type":"ContainerStarted","Data":"d7ce184b9be5ef70c3d384453788fa59c987424be0be0a893db9b5e993da7914"} Nov 02 06:25:10.597415 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:10.597357 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" podStartSLOduration=1.411383953 podStartE2EDuration="3.597342302s" totalImagesPullingTime="2.185958349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:25:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:25:10.597210408 +0000 UTC m=+487.045587182" watchObservedRunningTime="2026-11-02 06:25:10.597342302 +0000 UTC m=+487.045719055" Nov 02 06:25:10.824709 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:10.824662 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:11.590495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.590464 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf"] Nov 02 06:25:11.593972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.593951 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.695395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.695358 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 02 06:25:11.696441 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.696415 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf"] Nov 02 06:25:11.698762 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.698745 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.698844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.698822 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.698966 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.698856 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.698966 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.698943 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.699089 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.698969 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr8nv\" (UniqueName: \"kubernetes.io/projected/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-kube-api-access-dr8nv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.699154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.699072 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.800442 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.800403 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.800650 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.800456 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.800650 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.800524 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.800650 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.800550 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.800650 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.800592 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.800650 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.800625 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr8nv\" (UniqueName: \"kubernetes.io/projected/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-kube-api-access-dr8nv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.801030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.801004 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.801125 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.801027 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.801125 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.801065 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.802740 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.802722 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.803020 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.803003 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.811534 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.811512 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:11.869229 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.869170 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr8nv\" (UniqueName: \"kubernetes.io/projected/e1b38f9d-ab3b-4722-99b6-98ba6d9972f2-kube-api-access-dr8nv\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf\" (UID: \"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:11.925108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:11.925066 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:12.287793 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:12.287771 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf"] Nov 02 06:25:12.289904 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:25:12.289872 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1b38f9d_ab3b_4722_99b6_98ba6d9972f2.slice/crio-77bb769e7eb88adbbbe62743b3db42dbd36b0615067cb86f87629063ac99f7a9 WatchSource:0}: Error finding container 77bb769e7eb88adbbbe62743b3db42dbd36b0615067cb86f87629063ac99f7a9: Status 404 returned error can't find the container with id 77bb769e7eb88adbbbe62743b3db42dbd36b0615067cb86f87629063ac99f7a9 Nov 02 06:25:12.458630 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:12.458589 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" event={"ID":"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2","Type":"ContainerStarted","Data":"77bb769e7eb88adbbbe62743b3db42dbd36b0615067cb86f87629063ac99f7a9"} Nov 02 06:25:13.463619 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:13.463586 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" event={"ID":"e1b38f9d-ab3b-4722-99b6-98ba6d9972f2","Type":"ContainerStarted","Data":"486e19b31daecbaf1ab3c23a066ea72293ca1721397eb02c0ba4f6b11a4fafa3"} Nov 02 06:25:13.587412 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:13.587360 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podStartSLOduration=1.945223132 podStartE2EDuration="2.587345769s" totalImagesPullingTime="642.122637ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:25:13.584777208 +0000 UTC m=+490.033153966" watchObservedRunningTime="2026-11-02 06:25:13.587345769 +0000 UTC m=+490.035722553" Nov 02 06:25:13.925297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:13.925269 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:16.363845 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:16.363813 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:16.911258 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:16.911228 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv"] Nov 02 06:25:16.914391 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:16.914369 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:16.964233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:16.964206 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Nov 02 06:25:17.011897 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.011865 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv"] Nov 02 06:25:17.039164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.039124 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl6n6\" (UniqueName: \"kubernetes.io/projected/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-kube-api-access-fl6n6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.039342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.039171 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.039342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.039284 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.039342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.039321 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.039342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.039341 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.039513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.039421 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.056987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.056946 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:17.140224 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.140184 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.140430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.140239 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.140430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.140265 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.140430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.140309 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.140430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.140335 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl6n6\" (UniqueName: \"kubernetes.io/projected/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-kube-api-access-fl6n6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.140430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.140373 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.140691 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.140671 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.140779 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.140760 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.140828 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.140795 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.142594 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.142574 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.147801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.143103 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.195628 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.195601 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl6n6\" (UniqueName: \"kubernetes.io/projected/97ccb6cf-65c5-48e8-b65b-ad18ac981feb-kube-api-access-fl6n6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv\" (UID: \"97ccb6cf-65c5-48e8-b65b-ad18ac981feb\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.226645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.226613 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:17.592865 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.592793 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv"] Nov 02 06:25:17.596218 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:25:17.596187 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ccb6cf_65c5_48e8_b65b_ad18ac981feb.slice/crio-6581fc984f7757d1eee4e43fcedd7e6e58978cee1f0d89ac53aca09b0c40d1a4 WatchSource:0}: Error finding container 6581fc984f7757d1eee4e43fcedd7e6e58978cee1f0d89ac53aca09b0c40d1a4: Status 404 returned error can't find the container with id 6581fc984f7757d1eee4e43fcedd7e6e58978cee1f0d89ac53aca09b0c40d1a4 Nov 02 06:25:17.648063 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.648032 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:17.823990 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.823953 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:17.836514 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:17.836488 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:18.484516 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:18.484429 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" event={"ID":"97ccb6cf-65c5-48e8-b65b-ad18ac981feb","Type":"ContainerStarted","Data":"dc91675daf34d25bbec2487c46e63baa7a6c4210c116e8a5508a4afe0895062f"} Nov 02 06:25:18.484516 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:18.484490 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" event={"ID":"97ccb6cf-65c5-48e8-b65b-ad18ac981feb","Type":"ContainerStarted","Data":"6581fc984f7757d1eee4e43fcedd7e6e58978cee1f0d89ac53aca09b0c40d1a4"} Nov 02 06:25:18.495667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:18.495641 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-btbz5" Nov 02 06:25:18.728428 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:18.728372 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" podStartSLOduration=2.094409601 podStartE2EDuration="2.728357139s" totalImagesPullingTime="633.947538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:25:18.601832294 +0000 UTC m=+495.050209075" watchObservedRunningTime="2026-11-02 06:25:18.728357139 +0000 UTC m=+495.176733890" Nov 02 06:25:19.226944 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:19.226906 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:21.925932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:21.925894 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:21.938454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:21.938428 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:22.512609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:22.512582 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:25:24.195849 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.195812 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v"] Nov 02 06:25:24.201680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.201661 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.273202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.273173 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v"] Nov 02 06:25:24.298768 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.298747 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 02 06:25:24.299999 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.299979 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538b7b40-28f3-48e2-8172-f5198f6c63fb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.300055 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.300026 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.300173 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.300160 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.300215 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.300181 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.300215 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.300205 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.300278 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.300220 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8glgk\" (UniqueName: \"kubernetes.io/projected/538b7b40-28f3-48e2-8172-f5198f6c63fb-kube-api-access-8glgk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.338619 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.338586 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:24.401443 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.401406 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538b7b40-28f3-48e2-8172-f5198f6c63fb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.401609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.401460 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.401609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.401498 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.401609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.401513 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.401609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.401533 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.401609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.401553 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8glgk\" (UniqueName: \"kubernetes.io/projected/538b7b40-28f3-48e2-8172-f5198f6c63fb-kube-api-access-8glgk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.401989 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.401957 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.402104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.402041 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.402174 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.402105 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.403743 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.403716 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/538b7b40-28f3-48e2-8172-f5198f6c63fb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.403916 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.403896 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/538b7b40-28f3-48e2-8172-f5198f6c63fb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.457535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.457472 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8glgk\" (UniqueName: \"kubernetes.io/projected/538b7b40-28f3-48e2-8172-f5198f6c63fb-kube-api-access-8glgk\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-x888v\" (UID: \"538b7b40-28f3-48e2-8172-f5198f6c63fb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.514647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.514614 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:24.877936 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:24.877759 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v"] Nov 02 06:25:24.880490 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:25:24.880465 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538b7b40_28f3_48e2_8172_f5198f6c63fb.slice/crio-91d39611c25b2d63d2effb4ec47543753a5c0da35cf1a97d3d758c4948669ecb WatchSource:0}: Error finding container 91d39611c25b2d63d2effb4ec47543753a5c0da35cf1a97d3d758c4948669ecb: Status 404 returned error can't find the container with id 91d39611c25b2d63d2effb4ec47543753a5c0da35cf1a97d3d758c4948669ecb Nov 02 06:25:25.515063 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:25.515027 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" event={"ID":"538b7b40-28f3-48e2-8172-f5198f6c63fb","Type":"ContainerStarted","Data":"91d39611c25b2d63d2effb4ec47543753a5c0da35cf1a97d3d758c4948669ecb"} Nov 02 06:25:26.520824 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:26.520788 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" event={"ID":"538b7b40-28f3-48e2-8172-f5198f6c63fb","Type":"ContainerStarted","Data":"6c067c231424e089709fe9efaee74d94fb19f4a90dde51d2a2692df9498891bb"} Nov 02 06:25:26.630679 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:26.630608 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" podStartSLOduration=1.949483248 podStartE2EDuration="2.630586521s" totalImagesPullingTime="681.103273ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:25:26.630001325 +0000 UTC m=+503.078378101" watchObservedRunningTime="2026-11-02 06:25:26.630586521 +0000 UTC m=+503.078963278" Nov 02 06:25:27.226738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.226699 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:27.239063 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.239035 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:27.395631 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.395599 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9"] Nov 02 06:25:27.399168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.399150 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.448884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.448856 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 02 06:25:27.454286 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.454244 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9"] Nov 02 06:25:27.515239 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.515166 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:27.528713 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.528675 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.528713 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.528715 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74ff53da-f5b3-482e-a31a-d01941e33562-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.529248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.528738 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.529248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.528812 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.529248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.528851 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjfs7\" (UniqueName: \"kubernetes.io/projected/74ff53da-f5b3-482e-a31a-d01941e33562-kube-api-access-jjfs7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.529248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.528909 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.541094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.541049 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv" Nov 02 06:25:27.606761 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.606732 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:27.630028 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.629992 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.630198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.630038 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74ff53da-f5b3-482e-a31a-d01941e33562-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.630198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.630071 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.630198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.630115 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.630198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.630159 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjfs7\" (UniqueName: \"kubernetes.io/projected/74ff53da-f5b3-482e-a31a-d01941e33562-kube-api-access-jjfs7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.630397 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.630304 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.630512 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.630477 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.630598 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.630504 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.630731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.630709 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.632923 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.632900 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/74ff53da-f5b3-482e-a31a-d01941e33562-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.633168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.633152 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/74ff53da-f5b3-482e-a31a-d01941e33562-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.688815 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.688782 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjfs7\" (UniqueName: \"kubernetes.io/projected/74ff53da-f5b3-482e-a31a-d01941e33562-kube-api-access-jjfs7\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-m59z9\" (UID: \"74ff53da-f5b3-482e-a31a-d01941e33562\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:27.711660 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:27.711636 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:28.077379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:28.077352 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9"] Nov 02 06:25:28.079737 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:25:28.079711 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ff53da_f5b3_482e_a31a_d01941e33562.slice/crio-ffd4e54f0cf2e7997d85c937269d39a2f3107e486b67abf33b9679654d186b53 WatchSource:0}: Error finding container ffd4e54f0cf2e7997d85c937269d39a2f3107e486b67abf33b9679654d186b53: Status 404 returned error can't find the container with id ffd4e54f0cf2e7997d85c937269d39a2f3107e486b67abf33b9679654d186b53 Nov 02 06:25:28.534149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:28.534109 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" event={"ID":"74ff53da-f5b3-482e-a31a-d01941e33562","Type":"ContainerStarted","Data":"ffd4e54f0cf2e7997d85c937269d39a2f3107e486b67abf33b9679654d186b53"} Nov 02 06:25:29.539746 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:29.539703 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" event={"ID":"74ff53da-f5b3-482e-a31a-d01941e33562","Type":"ContainerStarted","Data":"e43c552e763e833204655af0c7837824dff424e98bd62836afe4f060cc8c77c4"} Nov 02 06:25:29.649055 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:29.648977 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" podStartSLOduration=2.003203023 podStartE2EDuration="2.648961612s" totalImagesPullingTime="645.758589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:25:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:25:29.648928115 +0000 UTC m=+506.097305067" watchObservedRunningTime="2026-11-02 06:25:29.648961612 +0000 UTC m=+506.097338369" Nov 02 06:25:29.712782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:29.712748 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:30.140555 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.140521 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22"] Nov 02 06:25:30.144922 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.144897 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.198274 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.198241 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22"] Nov 02 06:25:30.243472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.243441 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 02 06:25:30.254824 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.254792 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58816ff6-8725-4c1a-8b23-a9770d599670-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.254970 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.254875 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.254970 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.254928 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.255308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.254988 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.255308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.255185 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5msg\" (UniqueName: \"kubernetes.io/projected/58816ff6-8725-4c1a-8b23-a9770d599670-kube-api-access-k5msg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.255308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.255243 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.340577 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.340532 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:30.355638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.355615 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58816ff6-8725-4c1a-8b23-a9770d599670-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.355755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.355687 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.355755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.355706 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.355755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.355721 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.355755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.355739 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5msg\" (UniqueName: \"kubernetes.io/projected/58816ff6-8725-4c1a-8b23-a9770d599670-kube-api-access-k5msg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.355944 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.355895 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.356164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.356141 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.356270 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.356253 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.356378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.356360 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.357973 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.357954 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58816ff6-8725-4c1a-8b23-a9770d599670-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.358164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.358149 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58816ff6-8725-4c1a-8b23-a9770d599670-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.409946 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.409873 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5msg\" (UniqueName: \"kubernetes.io/projected/58816ff6-8725-4c1a-8b23-a9770d599670-kube-api-access-k5msg\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22\" (UID: \"58816ff6-8725-4c1a-8b23-a9770d599670\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.457819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.457792 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:30.821384 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:30.821358 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22"] Nov 02 06:25:30.823054 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:25:30.823030 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58816ff6_8725_4c1a_8b23_a9770d599670.slice/crio-4a32961c441de5948bc0dcdf0b77379b21bca786862958a5f7f992429cca6d55 WatchSource:0}: Error finding container 4a32961c441de5948bc0dcdf0b77379b21bca786862958a5f7f992429cca6d55: Status 404 returned error can't find the container with id 4a32961c441de5948bc0dcdf0b77379b21bca786862958a5f7f992429cca6d55 Nov 02 06:25:31.549257 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:31.549227 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" event={"ID":"58816ff6-8725-4c1a-8b23-a9770d599670","Type":"ContainerStarted","Data":"4a32961c441de5948bc0dcdf0b77379b21bca786862958a5f7f992429cca6d55"} Nov 02 06:25:32.554367 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:32.554331 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" event={"ID":"58816ff6-8725-4c1a-8b23-a9770d599670","Type":"ContainerStarted","Data":"bba5e7705b24196d9843d4d6385c461e159720b36184b992502ab9114cd25cc2"} Nov 02 06:25:32.666632 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:32.666569 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" podStartSLOduration=1.9899760560000002 podStartE2EDuration="2.66655403s" totalImagesPullingTime="676.577974ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:25:32.666391629 +0000 UTC m=+509.114768386" watchObservedRunningTime="2026-11-02 06:25:32.66655403 +0000 UTC m=+509.114930801" Nov 02 06:25:32.705890 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:25:32.705854 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:25:33.458304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:33.458262 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:33.714788 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:33.714703 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:34.514731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:34.514702 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:34.527159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:34.527127 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:34.572750 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:34.572721 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-x888v" Nov 02 06:25:36.973999 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:36.973967 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr"] Nov 02 06:25:36.977431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:36.977414 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.031620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.031588 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr"] Nov 02 06:25:37.076835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.076803 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 02 06:25:37.115626 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.115585 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.115626 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.115633 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25zx4\" (UniqueName: \"kubernetes.io/projected/92fa156c-b466-447a-a715-33b3cafe2a35-kube-api-access-25zx4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.115849 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.115653 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.115849 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.115705 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fa156c-b466-447a-a715-33b3cafe2a35-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.115928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.115867 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.115928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.115915 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.124267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.124234 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:37.216962 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.216926 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.217179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.216972 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.217179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.217012 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.217179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.217036 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25zx4\" (UniqueName: \"kubernetes.io/projected/92fa156c-b466-447a-a715-33b3cafe2a35-kube-api-access-25zx4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.217179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.217054 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.217179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.217098 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fa156c-b466-447a-a715-33b3cafe2a35-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.217461 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.217422 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.217461 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.217447 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.217607 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.217570 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.219401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.219374 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/92fa156c-b466-447a-a715-33b3cafe2a35-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.219549 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.219533 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/92fa156c-b466-447a-a715-33b3cafe2a35-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.271440 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.271374 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25zx4\" (UniqueName: \"kubernetes.io/projected/92fa156c-b466-447a-a715-33b3cafe2a35-kube-api-access-25zx4\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr\" (UID: \"92fa156c-b466-447a-a715-33b3cafe2a35\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.290110 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.290051 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:37.657149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.657122 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr"] Nov 02 06:25:37.659482 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:25:37.659455 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92fa156c_b466_447a_a715_33b3cafe2a35.slice/crio-24e3e9dc0ab61d9794bcdb836f8bf76dbd15f2224765179c976eb514a4cd3a31 WatchSource:0}: Error finding container 24e3e9dc0ab61d9794bcdb836f8bf76dbd15f2224765179c976eb514a4cd3a31: Status 404 returned error can't find the container with id 24e3e9dc0ab61d9794bcdb836f8bf76dbd15f2224765179c976eb514a4cd3a31 Nov 02 06:25:37.712027 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.711991 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:37.724809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:37.724780 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:38.586445 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:38.586346 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" event={"ID":"92fa156c-b466-447a-a715-33b3cafe2a35","Type":"ContainerStarted","Data":"5a04ac10531bd206d031ee0777d8da9a798c7fabb8bc00f2a33804eb607f411b"} Nov 02 06:25:38.586445 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:38.586405 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" event={"ID":"92fa156c-b466-447a-a715-33b3cafe2a35","Type":"ContainerStarted","Data":"24e3e9dc0ab61d9794bcdb836f8bf76dbd15f2224765179c976eb514a4cd3a31"} Nov 02 06:25:38.597644 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:38.597619 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-m59z9" Nov 02 06:25:38.696628 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:38.696578 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" podStartSLOduration=2.068582378 podStartE2EDuration="2.696556836s" totalImagesPullingTime="627.974458ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:25:38.695731585 +0000 UTC m=+515.144108367" watchObservedRunningTime="2026-11-02 06:25:38.696556836 +0000 UTC m=+515.144933591" Nov 02 06:25:39.290419 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:39.290376 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:40.458372 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:40.458335 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:40.470979 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:40.470950 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:40.606437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:40.606412 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22" Nov 02 06:25:43.080383 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:43.080346 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:43.269260 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:43.269228 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:25:45.685201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.685166 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b"] Nov 02 06:25:45.688268 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.688251 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.737655 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.737619 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hptbf\"" Nov 02 06:25:45.789194 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.789162 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.789350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.789221 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.789350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.789301 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d713b5c0-e038-4884-979d-d248b3546c50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.789566 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.789534 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.789679 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.789660 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.789743 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.789723 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d713b5c0-e038-4884-979d-d248b3546c50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.789827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.789813 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.789884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.789870 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjsx\" (UniqueName: \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-kube-api-access-7kjsx\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.789930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.789904 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.876129 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.876092 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b"] Nov 02 06:25:45.891062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.891031 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.891243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.891095 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.891243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.891139 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d713b5c0-e038-4884-979d-d248b3546c50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.891243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.891200 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.891406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.891254 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjsx\" (UniqueName: \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-kube-api-access-7kjsx\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.891406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.891283 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.891406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.891345 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.891406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.891396 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.891604 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.891429 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d713b5c0-e038-4884-979d-d248b3546c50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.892012 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.891981 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.892148 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.891994 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.892261 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.892235 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d713b5c0-e038-4884-979d-d248b3546c50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.892323 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.892290 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.892323 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.892291 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.893820 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.893795 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.894062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.894043 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d713b5c0-e038-4884-979d-d248b3546c50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.946174 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.946088 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjsx\" (UniqueName: \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-kube-api-access-7kjsx\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:45.946297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:45.946266 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-m478b\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:46.000326 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:46.000294 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:46.409044 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:46.409018 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b"] Nov 02 06:25:46.411268 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:25:46.411240 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd713b5c0_e038_4884_979d_d248b3546c50.slice/crio-2281810f2b7c09eeb77f128f5778ae428cdc1d63fab702c140de94f9a8985fb8 WatchSource:0}: Error finding container 2281810f2b7c09eeb77f128f5778ae428cdc1d63fab702c140de94f9a8985fb8: Status 404 returned error can't find the container with id 2281810f2b7c09eeb77f128f5778ae428cdc1d63fab702c140de94f9a8985fb8 Nov 02 06:25:46.620506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:46.620477 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" event={"ID":"d713b5c0-e038-4884-979d-d248b3546c50","Type":"ContainerStarted","Data":"2281810f2b7c09eeb77f128f5778ae428cdc1d63fab702c140de94f9a8985fb8"} Nov 02 06:25:47.290699 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:47.290668 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:47.304273 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:47.304242 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:47.636193 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:47.636115 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr" Nov 02 06:25:52.559982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:52.559947 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:25:52.560295 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:52.560018 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:25:52.560295 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:52.560046 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:25:52.648524 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:52.648491 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" event={"ID":"d713b5c0-e038-4884-979d-d248b3546c50","Type":"ContainerStarted","Data":"4a936b9100e13437c02297ed3b91ede21a693fac81930ff4046488870d53d65b"} Nov 02 06:25:52.779761 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:52.779705 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" podStartSLOduration=1.633676641 podStartE2EDuration="7.779690321s" totalImagesPullingTime="6.14601368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:25:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:25:52.778329175 +0000 UTC m=+529.226705958" watchObservedRunningTime="2026-11-02 06:25:52.779690321 +0000 UTC m=+529.228067131" Nov 02 06:25:53.004113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:53.002837 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:25:53.004113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:53.003016 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-m478b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 06:25:53.004113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:53.003067 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" podUID="d713b5c0-e038-4884-979d-d248b3546c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 06:25:54.005101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:54.004510 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-m478b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 06:25:54.005101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:54.004587 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" podUID="d713b5c0-e038-4884-979d-d248b3546c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 06:25:55.006101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:55.002459 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-m478b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 06:25:55.006101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:55.002520 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" podUID="d713b5c0-e038-4884-979d-d248b3546c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 06:25:56.001512 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:56.001472 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-m478b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 06:25:56.001816 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:56.001786 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" podUID="d713b5c0-e038-4884-979d-d248b3546c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 06:25:57.005100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:57.003441 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-m478b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 06:25:57.005100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:57.003504 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" podUID="d713b5c0-e038-4884-979d-d248b3546c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 06:25:58.003179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:58.003142 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-m478b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 06:25:58.003458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:58.003429 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" podUID="d713b5c0-e038-4884-979d-d248b3546c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 06:25:59.001261 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:59.001228 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-m478b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Nov 02 06:25:59.003568 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:25:59.001277 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" podUID="d713b5c0-e038-4884-979d-d248b3546c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Nov 02 06:26:00.752461 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:00.752431 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:26:00.752962 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:00.752819 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:26:00.753588 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:00.753566 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:26:02.707575 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:26:02.707539 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:26:02.736151 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.736126 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6"] Nov 02 06:26:02.739517 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.739502 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:02.764334 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.764304 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-zk5j6\" (UID: \"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:02.764492 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.764472 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7s7g\" (UniqueName: \"kubernetes.io/projected/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-kube-api-access-l7s7g\") pod \"ai-gateway-controller-5f6dd4f69f-zk5j6\" (UID: \"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:02.786170 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.786136 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:02.842112 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.842012 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-d9228\"" Nov 02 06:26:02.843695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.842397 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 02 06:26:02.843844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.843755 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 06:26:02.843844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.843798 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 06:26:02.844201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.844175 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 06:26:02.844337 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.844314 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6"] Nov 02 06:26:02.865345 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.865321 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7s7g\" (UniqueName: \"kubernetes.io/projected/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-kube-api-access-l7s7g\") pod \"ai-gateway-controller-5f6dd4f69f-zk5j6\" (UID: \"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:02.865458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.865374 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-zk5j6\" (UID: \"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:02.867609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.867583 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-metrics-certs\") pod \"ai-gateway-controller-5f6dd4f69f-zk5j6\" (UID: \"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:02.919667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.919638 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7s7g\" (UniqueName: \"kubernetes.io/projected/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-kube-api-access-l7s7g\") pod \"ai-gateway-controller-5f6dd4f69f-zk5j6\" (UID: \"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca\") " pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:02.966084 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:02.966053 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:03.051712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:03.051676 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:03.415051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:03.415025 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6"] Nov 02 06:26:03.417187 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:26:03.417154 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaebbbc1_a91d_42d5_9a88_3f7562e4c6ca.slice/crio-2a1a93d2243ee5a467203bd27e7461a7ab5f679449ff9b654150975e8cb90ee4 WatchSource:0}: Error finding container 2a1a93d2243ee5a467203bd27e7461a7ab5f679449ff9b654150975e8cb90ee4: Status 404 returned error can't find the container with id 2a1a93d2243ee5a467203bd27e7461a7ab5f679449ff9b654150975e8cb90ee4 Nov 02 06:26:03.692111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:03.692020 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" event={"ID":"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca","Type":"ContainerStarted","Data":"2a1a93d2243ee5a467203bd27e7461a7ab5f679449ff9b654150975e8cb90ee4"} Nov 02 06:26:09.241869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:09.241834 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 02 06:26:09.716856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:09.716818 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" event={"ID":"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca","Type":"ContainerStarted","Data":"9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43"} Nov 02 06:26:09.717035 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:09.716891 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:09.825895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:09.825846 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" podStartSLOduration=2.051865796 podStartE2EDuration="7.825833294s" totalImagesPullingTime="5.773967498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:26:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:26:09.825448224 +0000 UTC m=+546.273824980" watchObservedRunningTime="2026-11-02 06:26:09.825833294 +0000 UTC m=+546.274210077" Nov 02 06:26:13.105046 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.105015 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:13.310636 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.310606 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:13.436536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.436502 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh"] Nov 02 06:26:13.440146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.440130 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" Nov 02 06:26:13.539164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.539129 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 02 06:26:13.539376 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.539353 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-n62bv\"" Nov 02 06:26:13.541449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.541425 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh"] Nov 02 06:26:13.563570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.563531 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf7m9\" (UniqueName: \"kubernetes.io/projected/81beb271-42bd-4b5e-999f-686a5020dee6-kube-api-access-hf7m9\") pod \"payload-pre-processing-7b98bb678c-9tcjh\" (UID: \"81beb271-42bd-4b5e-999f-686a5020dee6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" Nov 02 06:26:13.563750 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.563725 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81beb271-42bd-4b5e-999f-686a5020dee6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9tcjh\" (UID: \"81beb271-42bd-4b5e-999f-686a5020dee6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" Nov 02 06:26:13.603551 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.603518 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lfnlg"] Nov 02 06:26:13.607481 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.607460 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" Nov 02 06:26:13.664827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.664794 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81beb271-42bd-4b5e-999f-686a5020dee6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9tcjh\" (UID: \"81beb271-42bd-4b5e-999f-686a5020dee6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" Nov 02 06:26:13.665011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.664837 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97699518-7b61-4b41-a574-ad41aefce208-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-lfnlg\" (UID: \"97699518-7b61-4b41-a574-ad41aefce208\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" Nov 02 06:26:13.665011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.664942 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf7m9\" (UniqueName: \"kubernetes.io/projected/81beb271-42bd-4b5e-999f-686a5020dee6-kube-api-access-hf7m9\") pod \"payload-pre-processing-7b98bb678c-9tcjh\" (UID: \"81beb271-42bd-4b5e-999f-686a5020dee6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" Nov 02 06:26:13.665175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.665013 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjxlk\" (UniqueName: \"kubernetes.io/projected/97699518-7b61-4b41-a574-ad41aefce208-kube-api-access-sjxlk\") pod \"payload-processing-55fbbbf9dc-lfnlg\" (UID: \"97699518-7b61-4b41-a574-ad41aefce208\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" Nov 02 06:26:13.665629 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.665594 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81beb271-42bd-4b5e-999f-686a5020dee6-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9tcjh\" (UID: \"81beb271-42bd-4b5e-999f-686a5020dee6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" Nov 02 06:26:13.668714 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.668687 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lfnlg"] Nov 02 06:26:13.724464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.724386 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf7m9\" (UniqueName: \"kubernetes.io/projected/81beb271-42bd-4b5e-999f-686a5020dee6-kube-api-access-hf7m9\") pod \"payload-pre-processing-7b98bb678c-9tcjh\" (UID: \"81beb271-42bd-4b5e-999f-686a5020dee6\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" Nov 02 06:26:13.752202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.752174 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" Nov 02 06:26:13.766593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.766554 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjxlk\" (UniqueName: \"kubernetes.io/projected/97699518-7b61-4b41-a574-ad41aefce208-kube-api-access-sjxlk\") pod \"payload-processing-55fbbbf9dc-lfnlg\" (UID: \"97699518-7b61-4b41-a574-ad41aefce208\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" Nov 02 06:26:13.766746 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.766655 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97699518-7b61-4b41-a574-ad41aefce208-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-lfnlg\" (UID: \"97699518-7b61-4b41-a574-ad41aefce208\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" Nov 02 06:26:13.767618 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.767566 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97699518-7b61-4b41-a574-ad41aefce208-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-lfnlg\" (UID: \"97699518-7b61-4b41-a574-ad41aefce208\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" Nov 02 06:26:13.821954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.821927 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjxlk\" (UniqueName: \"kubernetes.io/projected/97699518-7b61-4b41-a574-ad41aefce208-kube-api-access-sjxlk\") pod \"payload-processing-55fbbbf9dc-lfnlg\" (UID: \"97699518-7b61-4b41-a574-ad41aefce208\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" Nov 02 06:26:13.922035 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:13.921948 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" Nov 02 06:26:14.113638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:14.113610 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh"] Nov 02 06:26:14.114042 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:26:14.114019 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81beb271_42bd_4b5e_999f_686a5020dee6.slice/crio-ccb5fe85aebe3dad227703d8418a041766d7bd605d3df3edd8889e6003c7c5e2 WatchSource:0}: Error finding container ccb5fe85aebe3dad227703d8418a041766d7bd605d3df3edd8889e6003c7c5e2: Status 404 returned error can't find the container with id ccb5fe85aebe3dad227703d8418a041766d7bd605d3df3edd8889e6003c7c5e2 Nov 02 06:26:14.292250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:14.292229 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-lfnlg"] Nov 02 06:26:14.293361 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:26:14.293316 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97699518_7b61_4b41_a574_ad41aefce208.slice/crio-19414300b3ee2020f18997d8889bce8b4a8766236009a40b0173ddd9af89ab5e WatchSource:0}: Error finding container 19414300b3ee2020f18997d8889bce8b4a8766236009a40b0173ddd9af89ab5e: Status 404 returned error can't find the container with id 19414300b3ee2020f18997d8889bce8b4a8766236009a40b0173ddd9af89ab5e Nov 02 06:26:14.741543 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:14.741505 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" event={"ID":"81beb271-42bd-4b5e-999f-686a5020dee6","Type":"ContainerStarted","Data":"ccb5fe85aebe3dad227703d8418a041766d7bd605d3df3edd8889e6003c7c5e2"} Nov 02 06:26:14.742938 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:14.742910 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" event={"ID":"97699518-7b61-4b41-a574-ad41aefce208","Type":"ContainerStarted","Data":"19414300b3ee2020f18997d8889bce8b4a8766236009a40b0173ddd9af89ab5e"} Nov 02 06:26:16.641469 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:16.641437 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:17.619638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:17.619560 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:17.759381 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:17.759343 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" event={"ID":"97699518-7b61-4b41-a574-ad41aefce208","Type":"ContainerStarted","Data":"013a19c3b1dbc541c24c220e5ea67f61d5b08e65df0ca761e7489d5c5dfa4ea7"} Nov 02 06:26:17.759804 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:17.759496 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" Nov 02 06:26:17.760863 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:17.760840 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" event={"ID":"81beb271-42bd-4b5e-999f-686a5020dee6","Type":"ContainerStarted","Data":"b160b60e5ba74de36af08cdf0fadac16637f91359a21d40a7ec94c10b79bc848"} Nov 02 06:26:17.760963 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:17.760895 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" Nov 02 06:26:17.825352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:17.825325 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:17.871564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:17.871482 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" podStartSLOduration=1.740212627 podStartE2EDuration="4.871470077s" totalImagesPullingTime="3.13125745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:26:17.870380064 +0000 UTC m=+554.318756832" watchObservedRunningTime="2026-11-02 06:26:17.871470077 +0000 UTC m=+554.319846832" Nov 02 06:26:17.978746 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:17.978688 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" podStartSLOduration=1.770713134 podStartE2EDuration="4.97867264s" totalImagesPullingTime="3.207959506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:26:17.977935867 +0000 UTC m=+554.426312641" watchObservedRunningTime="2026-11-02 06:26:17.97867264 +0000 UTC m=+554.427049396" Nov 02 06:26:20.723479 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:20.723450 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:28.769918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:28.769881 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9tcjh" Nov 02 06:26:28.770461 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:28.769952 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-lfnlg" Nov 02 06:26:32.706643 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:26:32.706606 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:26:37.802373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:37.802339 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6"] Nov 02 06:26:37.802815 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:37.802535 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" podUID="caebbbc1-a91d-42d5-9a88-3f7562e4c6ca" containerName="manager" containerID="cri-o://9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43" gracePeriod=30 Nov 02 06:26:38.046189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.046161 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:38.187268 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.187242 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-metrics-certs\") pod \"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca\" (UID: \"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca\") " Nov 02 06:26:38.187448 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.187370 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-kube-api-access-l7s7g\" (UniqueName: \"kubernetes.io/projected/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-kube-api-access-l7s7g\") pod \"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca\" (UID: \"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca\") " Nov 02 06:26:38.189439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.189413 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-kube-api-access-l7s7g" pod "caebbbc1-a91d-42d5-9a88-3f7562e4c6ca" (UID: "caebbbc1-a91d-42d5-9a88-3f7562e4c6ca"). InnerVolumeSpecName "kube-api-access-l7s7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:26:38.189525 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.189418 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-metrics-certs" pod "caebbbc1-a91d-42d5-9a88-3f7562e4c6ca" (UID: "caebbbc1-a91d-42d5-9a88-3f7562e4c6ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:26:38.288711 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.288668 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:26:38.288711 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.288708 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7s7g\" (UniqueName: \"kubernetes.io/projected/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca-kube-api-access-l7s7g\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:26:38.849157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.849124 5025 generic.go:356] "Generic (PLEG): container finished" podID="caebbbc1-a91d-42d5-9a88-3f7562e4c6ca" containerID="9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43" exitCode=0 Nov 02 06:26:38.849598 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.849167 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" Nov 02 06:26:38.849598 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.849206 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" event={"ID":"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca","Type":"ContainerDied","Data":"9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43"} Nov 02 06:26:38.849598 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.849255 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6" event={"ID":"caebbbc1-a91d-42d5-9a88-3f7562e4c6ca","Type":"ContainerDied","Data":"2a1a93d2243ee5a467203bd27e7461a7ab5f679449ff9b654150975e8cb90ee4"} Nov 02 06:26:38.849598 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.849277 5025 scope.go:122] "RemoveContainer" containerID="9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43" Nov 02 06:26:38.857620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.857603 5025 scope.go:122] "RemoveContainer" containerID="9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43" Nov 02 06:26:38.857876 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:26:38.857849 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43\": container with ID starting with 9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43 not found: ID does not exist" containerID="9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43" Nov 02 06:26:38.857921 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:38.857882 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43"} err="failed to get container status \"9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43\": rpc error: code = NotFound desc = could not find container \"9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43\": container with ID starting with 9788732d5b699f4eb57d0a7774439e1db7e99af2a5871c7755835861bb016d43 not found: ID does not exist" Nov 02 06:26:39.004779 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:39.004747 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6"] Nov 02 06:26:39.007793 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:39.007768 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5f6dd4f69f-zk5j6"] Nov 02 06:26:39.577810 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:39.577779 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:39.667087 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:39.667045 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caebbbc1-a91d-42d5-9a88-3f7562e4c6ca" path="/var/lib/kubelet/pods/caebbbc1-a91d-42d5-9a88-3f7562e4c6ca/volumes" Nov 02 06:26:40.050441 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:40.050393 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:40.322918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:40.322837 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:41.596736 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:41.596698 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:41.776308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:41.776257 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:43.337189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:43.337156 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:43.642603 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:43.642531 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:26:46.134450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.134415 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr"] Nov 02 06:26:46.134858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.134768 5025 cpu_manager.go:414] "containerMap: removing container" podUID="caebbbc1-a91d-42d5-9a88-3f7562e4c6ca" containerName="manager" Nov 02 06:26:46.134858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.134779 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caebbbc1-a91d-42d5-9a88-3f7562e4c6ca" containerName="manager" Nov 02 06:26:46.134858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.134851 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="caebbbc1-a91d-42d5-9a88-3f7562e4c6ca" containerName="manager" Nov 02 06:26:46.137897 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.137880 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.236123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.236088 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr"] Nov 02 06:26:46.257043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.257011 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec38b35a-07e4-4a82-9444-727ede78e928-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.257216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.257060 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkjs8\" (UniqueName: \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-kube-api-access-nkjs8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.257216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.257193 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec38b35a-07e4-4a82-9444-727ede78e928-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.257291 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.257231 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.257291 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.257271 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.257356 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.257291 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.257356 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.257348 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.257426 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.257406 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.257463 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.257437 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.358782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.358745 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkjs8\" (UniqueName: \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-kube-api-access-nkjs8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.358782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.358783 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec38b35a-07e4-4a82-9444-727ede78e928-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.358806 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.358834 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.358852 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.358870 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.358889 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.358916 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.358968 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec38b35a-07e4-4a82-9444-727ede78e928-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359485 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.359350 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359485 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.359391 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359485 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.359444 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.359524 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.359706 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.359663 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec38b35a-07e4-4a82-9444-727ede78e928-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.361228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.361199 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.361482 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.361460 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec38b35a-07e4-4a82-9444-727ede78e928-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.414462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.414439 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.414573 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.414510 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkjs8\" (UniqueName: \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-kube-api-access-nkjs8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-zftfr\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.450307 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.450284 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:46.861493 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.861467 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr"] Nov 02 06:26:46.863459 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:26:46.863433 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec38b35a_07e4_4a82_9444_727ede78e928.slice/crio-2e43b33712a93b443430fa01ed19b6fee0032c43faab5f772551aad6edd81667 WatchSource:0}: Error finding container 2e43b33712a93b443430fa01ed19b6fee0032c43faab5f772551aad6edd81667: Status 404 returned error can't find the container with id 2e43b33712a93b443430fa01ed19b6fee0032c43faab5f772551aad6edd81667 Nov 02 06:26:46.865616 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.865583 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:26:46.865715 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.865645 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:26:46.865715 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.865704 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:26:46.885042 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:46.885013 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" event={"ID":"ec38b35a-07e4-4a82-9444-727ede78e928","Type":"ContainerStarted","Data":"2e43b33712a93b443430fa01ed19b6fee0032c43faab5f772551aad6edd81667"} Nov 02 06:26:47.911621 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:47.911578 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" event={"ID":"ec38b35a-07e4-4a82-9444-727ede78e928","Type":"ContainerStarted","Data":"3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c"} Nov 02 06:26:48.122200 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:48.122152 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" podStartSLOduration=2.122139266 podStartE2EDuration="2.122139266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:26:48.074760056 +0000 UTC m=+584.523136855" watchObservedRunningTime="2026-11-02 06:26:48.122139266 +0000 UTC m=+584.570516022" Nov 02 06:26:48.451253 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:48.451214 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:48.452774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:48.452747 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-zftfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:26:48.452905 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:48.452800 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 06:26:49.451565 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:49.451524 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-zftfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:26:49.451960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:49.451577 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 06:26:50.450870 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:50.450831 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-zftfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:26:50.451032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:50.450885 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 06:26:51.451268 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:51.451235 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-zftfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:26:51.451660 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:51.451288 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 06:26:52.451283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:52.451247 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-zftfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:26:52.451661 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:52.451300 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 06:26:53.451179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:53.451139 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-zftfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Nov 02 06:26:53.451370 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:53.451200 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Nov 02 06:26:55.096190 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:55.096160 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:55.096678 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:55.096571 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:26:55.097505 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:26:55.097462 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:27:02.706310 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:27:02.706274 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:27:11.110028 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.109992 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:11.296983 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.296947 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:11.803166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.803137 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq"] Nov 02 06:27:11.810031 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.810010 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:11.867245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.867220 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-581608-openshift-default-dockercfg-bm4lm\"" Nov 02 06:27:11.911327 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.911298 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq"] Nov 02 06:27:11.919857 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.919835 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqlrg\" (UniqueName: \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-kube-api-access-xqlrg\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:11.919950 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.919881 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:11.919950 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.919903 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:11.920026 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.920009 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60924270-4daa-44e7-a026-cc44a28d77eb-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:11.920066 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.920054 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60924270-4daa-44e7-a026-cc44a28d77eb-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:11.920138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.920122 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:11.920178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.920167 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:11.920219 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.920186 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:11.920254 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:11.920221 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021038 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021008 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021038 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021043 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021097 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60924270-4daa-44e7-a026-cc44a28d77eb-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021152 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60924270-4daa-44e7-a026-cc44a28d77eb-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021179 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021214 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021237 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021289 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021344 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqlrg\" (UniqueName: \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-kube-api-access-xqlrg\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021495 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021729 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021697 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021843 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021814 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021897 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021840 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.021987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.021969 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60924270-4daa-44e7-a026-cc44a28d77eb-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.023509 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.023489 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.023853 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.023823 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60924270-4daa-44e7-a026-cc44a28d77eb-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.075943 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.075885 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.076036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.075958 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqlrg\" (UniqueName: \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-kube-api-access-xqlrg\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.122290 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.122263 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:12.538125 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.538099 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq"] Nov 02 06:27:12.539565 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:27:12.539539 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60924270_4daa_44e7_a026_cc44a28d77eb.slice/crio-32acf9b5e1a2dc0c36966963373400f9b940b05aefbc4428a68ff6430fd1ee4f WatchSource:0}: Error finding container 32acf9b5e1a2dc0c36966963373400f9b940b05aefbc4428a68ff6430fd1ee4f: Status 404 returned error can't find the container with id 32acf9b5e1a2dc0c36966963373400f9b940b05aefbc4428a68ff6430fd1ee4f Nov 02 06:27:12.541826 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.541786 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:12.541908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.541867 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:12.541952 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.541908 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:12.697748 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.697711 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz"] Nov 02 06:27:12.704514 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.704481 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.755551 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.755499 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-dcf9c3-openshift-default-dockercfg-7kbz6\"" Nov 02 06:27:12.757432 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.757406 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz"] Nov 02 06:27:12.829281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.829206 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.829281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.829251 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.829281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.829281 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.829557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.829433 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.829557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.829487 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.829666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.829566 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.829666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.829603 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.829666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.829645 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.829766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.829700 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xgjw\" (UniqueName: \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-kube-api-access-7xgjw\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.927304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.927255 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:12.930370 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930352 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930453 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930384 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930453 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930408 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930453 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930436 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930468 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930493 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xgjw\" (UniqueName: \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-kube-api-access-7xgjw\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930528 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930567 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930775 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930609 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930775 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930746 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930811 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930938 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930908 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.930985 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.930955 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.931399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.931374 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.932799 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.932779 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.933104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.933064 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.990403 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.990365 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:12.992365 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:12.992345 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xgjw\" (UniqueName: \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-kube-api-access-7xgjw\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:13.019628 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.019599 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:13.026508 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.026479 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" event={"ID":"60924270-4daa-44e7-a026-cc44a28d77eb","Type":"ContainerStarted","Data":"75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf"} Nov 02 06:27:13.026712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.026685 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" event={"ID":"60924270-4daa-44e7-a026-cc44a28d77eb","Type":"ContainerStarted","Data":"32acf9b5e1a2dc0c36966963373400f9b940b05aefbc4428a68ff6430fd1ee4f"} Nov 02 06:27:13.133373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.133146 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:13.144118 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.144047 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:13.147662 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.147637 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:13.257967 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.257917 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" podStartSLOduration=2.257900342 podStartE2EDuration="2.257900342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:13.14501102 +0000 UTC m=+609.593387802" watchObservedRunningTime="2026-11-02 06:27:13.257900342 +0000 UTC m=+609.706277095" Nov 02 06:27:13.350465 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.350398 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:13.446631 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.446596 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz"] Nov 02 06:27:13.452811 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:27:13.452781 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9c4a77_9cc7_4478_966e_e0762e9c5c97.slice/crio-633f313d498f630b7aff4a1d5cef07ff6ee0af8defb8aed441ea14ba130913d5 WatchSource:0}: Error finding container 633f313d498f630b7aff4a1d5cef07ff6ee0af8defb8aed441ea14ba130913d5: Status 404 returned error can't find the container with id 633f313d498f630b7aff4a1d5cef07ff6ee0af8defb8aed441ea14ba130913d5 Nov 02 06:27:13.455742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.455704 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:13.455823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.455779 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:13.455823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.455810 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:13.553469 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.553420 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:13.725866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:13.725782 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:14.032495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:14.032393 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" event={"ID":"6d9c4a77-9cc7-4478-966e-e0762e9c5c97","Type":"ContainerStarted","Data":"be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8"} Nov 02 06:27:14.032495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:14.032450 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" event={"ID":"6d9c4a77-9cc7-4478-966e-e0762e9c5c97","Type":"ContainerStarted","Data":"633f313d498f630b7aff4a1d5cef07ff6ee0af8defb8aed441ea14ba130913d5"} Nov 02 06:27:14.032728 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:14.032708 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:14.033579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:14.033559 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:14.152947 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:14.152900 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" podStartSLOduration=2.152887042 podStartE2EDuration="2.152887042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:14.150663652 +0000 UTC m=+610.599040407" watchObservedRunningTime="2026-11-02 06:27:14.152887042 +0000 UTC m=+610.601263800" Nov 02 06:27:15.020318 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:15.020285 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:15.026167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:15.026141 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:15.037662 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:15.037635 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:15.038583 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:15.038566 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:27:16.084301 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:16.084265 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:16.277747 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:16.277716 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:16.869310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:16.869258 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:17.132567 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:17.132463 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:17.440711 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:17.440680 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:17.879867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:17.879790 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:18.088874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:18.088827 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:19.514178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:19.514144 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:19.711665 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:19.711628 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:19.891729 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:19.891627 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:20.185256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.185230 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:20.369430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.369399 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx"] Nov 02 06:27:20.372964 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.372947 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 06:27:20.386838 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.386808 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:20.426502 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.426442 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx"] Nov 02 06:27:20.471004 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.470924 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-cdfce9\"" Nov 02 06:27:20.471004 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.470979 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-cdfce9-dockercfg-2k699\"" Nov 02 06:27:20.515995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.515965 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx\" (UID: \"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 06:27:20.516356 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.515999 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lrb\" (UniqueName: \"kubernetes.io/projected/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-kube-api-access-d6lrb\") pod \"payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx\" (UID: \"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 06:27:20.519788 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.519763 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8"] Nov 02 06:27:20.523053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.523036 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 06:27:20.566850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.566821 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8"] Nov 02 06:27:20.616844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.616813 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pk2x\" (UniqueName: \"kubernetes.io/projected/b9222860-abff-46e2-b001-1de0837f0213-kube-api-access-7pk2x\") pod \"payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8\" (UID: \"b9222860-abff-46e2-b001-1de0837f0213\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 06:27:20.617014 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.616941 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9222860-abff-46e2-b001-1de0837f0213-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8\" (UID: \"b9222860-abff-46e2-b001-1de0837f0213\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 06:27:20.617106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.617009 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx\" (UID: \"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 06:27:20.617106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.617046 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lrb\" (UniqueName: \"kubernetes.io/projected/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-kube-api-access-d6lrb\") pod \"payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx\" (UID: \"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 06:27:20.617741 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.617723 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx\" (UID: \"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 06:27:20.678471 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.678442 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lrb\" (UniqueName: \"kubernetes.io/projected/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-kube-api-access-d6lrb\") pod \"payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx\" (UID: \"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 06:27:20.685136 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.685110 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 06:27:20.718492 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.718462 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pk2x\" (UniqueName: \"kubernetes.io/projected/b9222860-abff-46e2-b001-1de0837f0213-kube-api-access-7pk2x\") pod \"payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8\" (UID: \"b9222860-abff-46e2-b001-1de0837f0213\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 06:27:20.718635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.718536 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9222860-abff-46e2-b001-1de0837f0213-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8\" (UID: \"b9222860-abff-46e2-b001-1de0837f0213\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 06:27:20.719351 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.719332 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9222860-abff-46e2-b001-1de0837f0213-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8\" (UID: \"b9222860-abff-46e2-b001-1de0837f0213\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 06:27:20.726152 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.726012 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:20.774586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.774554 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pk2x\" (UniqueName: \"kubernetes.io/projected/b9222860-abff-46e2-b001-1de0837f0213-kube-api-access-7pk2x\") pod \"payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8\" (UID: \"b9222860-abff-46e2-b001-1de0837f0213\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 06:27:20.835418 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:20.835387 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 06:27:21.028001 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:21.027923 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:21.049516 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:21.049496 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx"] Nov 02 06:27:21.051481 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:27:21.051454 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b1f206_c4f2_40bb_a1a7_ce97a7cfd1d2.slice/crio-1410653f82723759447c5b3a395be0b0d33f0b84c41c8211e2f091d71082bfa3 WatchSource:0}: Error finding container 1410653f82723759447c5b3a395be0b0d33f0b84c41c8211e2f091d71082bfa3: Status 404 returned error can't find the container with id 1410653f82723759447c5b3a395be0b0d33f0b84c41c8211e2f091d71082bfa3 Nov 02 06:27:21.069993 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:21.069964 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" event={"ID":"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2","Type":"ContainerStarted","Data":"1410653f82723759447c5b3a395be0b0d33f0b84c41c8211e2f091d71082bfa3"} Nov 02 06:27:21.195899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:21.195869 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8"] Nov 02 06:27:21.197822 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:27:21.197798 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9222860_abff_46e2_b001_1de0837f0213.slice/crio-5ee059c99fe3ce8df34e58894894e2d4b116f0b20441e8b6eac5c65b02e96f44 WatchSource:0}: Error finding container 5ee059c99fe3ce8df34e58894894e2d4b116f0b20441e8b6eac5c65b02e96f44: Status 404 returned error can't find the container with id 5ee059c99fe3ce8df34e58894894e2d4b116f0b20441e8b6eac5c65b02e96f44 Nov 02 06:27:22.068365 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.068335 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9"] Nov 02 06:27:22.071709 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.071688 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.076589 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.076559 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" event={"ID":"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2","Type":"ContainerStarted","Data":"bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76"} Nov 02 06:27:22.076726 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.076706 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 06:27:22.078236 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.078213 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" event={"ID":"b9222860-abff-46e2-b001-1de0837f0213","Type":"ContainerStarted","Data":"c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1"} Nov 02 06:27:22.078391 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.078370 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" event={"ID":"b9222860-abff-46e2-b001-1de0837f0213","Type":"ContainerStarted","Data":"5ee059c99fe3ce8df34e58894894e2d4b116f0b20441e8b6eac5c65b02e96f44"} Nov 02 06:27:22.078496 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.078405 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 06:27:22.171302 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.171267 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-581608-serving-cert\"" Nov 02 06:27:22.172795 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.172776 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9"] Nov 02 06:27:22.231882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.231851 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4trcw\" (UniqueName: \"kubernetes.io/projected/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-kube-api-access-4trcw\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.232111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.232065 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.232194 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.232151 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-maas-api-tls\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.232250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.232231 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-metrics-certs\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.292359 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.292313 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" podStartSLOduration=2.292300717 podStartE2EDuration="2.292300717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:22.291302513 +0000 UTC m=+618.739679269" watchObservedRunningTime="2026-11-02 06:27:22.292300717 +0000 UTC m=+618.740677488" Nov 02 06:27:22.304279 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.304256 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:22.333543 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.333483 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.333543 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.333518 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-maas-api-tls\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.333543 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.333542 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-metrics-certs\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.333710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.333588 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4trcw\" (UniqueName: \"kubernetes.io/projected/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-kube-api-access-4trcw\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.333710 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:27:22.333649 5025 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-serving-cert: secret "maas-api-e2e-worker-w2-581608-serving-cert" not found Nov 02 06:27:22.333786 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:27:22.333738 5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-maas-api-tls podName:2dfb3a3f-ea26-4cad-a870-c5f4c733d480 nodeName:}" failed. No retries permitted until 2026-11-02 06:27:22.833716864 +0000 UTC m=+619.282093613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-maas-api-tls") pod "maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" (UID: "2dfb3a3f-ea26-4cad-a870-c5f4c733d480") : secret "maas-api-e2e-worker-w2-581608-serving-cert" not found Nov 02 06:27:22.334193 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.334171 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.336055 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.336036 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-metrics-certs\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.388216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.388195 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4trcw\" (UniqueName: \"kubernetes.io/projected/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-kube-api-access-4trcw\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.399936 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.399900 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" podStartSLOduration=2.399889471 podStartE2EDuration="2.399889471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:22.399284072 +0000 UTC m=+618.847660829" watchObservedRunningTime="2026-11-02 06:27:22.399889471 +0000 UTC m=+618.848266226" Nov 02 06:27:22.501404 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.501374 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:22.710193 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.710165 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:22.838070 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.838034 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-maas-api-tls\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.840230 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.840203 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-maas-api-tls\") pod \"maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:22.984727 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:22.984637 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:23.352800 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:23.352688 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9"] Nov 02 06:27:23.355431 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:27:23.355400 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dfb3a3f_ea26_4cad_a870_c5f4c733d480.slice/crio-c84912db04f242aef2152dc8eb6522d70e18e6fbe7d203506a9952918bd1db27 WatchSource:0}: Error finding container c84912db04f242aef2152dc8eb6522d70e18e6fbe7d203506a9952918bd1db27: Status 404 returned error can't find the container with id c84912db04f242aef2152dc8eb6522d70e18e6fbe7d203506a9952918bd1db27 Nov 02 06:27:23.356798 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:23.356781 5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:27:24.088062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:24.088027 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" event={"ID":"2dfb3a3f-ea26-4cad-a870-c5f4c733d480","Type":"ContainerStarted","Data":"c84912db04f242aef2152dc8eb6522d70e18e6fbe7d203506a9952918bd1db27"} Nov 02 06:27:24.608362 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:24.608281 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:24.794892 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:24.794855 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:25.094586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.094540 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" event={"ID":"2dfb3a3f-ea26-4cad-a870-c5f4c733d480","Type":"ContainerStarted","Data":"d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e"} Nov 02 06:27:25.094785 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.094637 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:25.126603 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.126565 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2"] Nov 02 06:27:25.131417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.131390 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 06:27:25.181636 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.181605 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2"] Nov 02 06:27:25.199535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.199509 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-581608\"" Nov 02 06:27:25.199535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.199517 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-581608-dockercfg-j8dbg\"" Nov 02 06:27:25.232722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.232680 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" podStartSLOduration=2.312530784 podStartE2EDuration="3.232665752s" totalImagesPullingTime="920.134968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:25.231628762 +0000 UTC m=+621.680005519" watchObservedRunningTime="2026-11-02 06:27:25.232665752 +0000 UTC m=+621.681042509" Nov 02 06:27:25.267527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.267499 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f86sc\" (UniqueName: \"kubernetes.io/projected/49f36379-dc5e-434c-84be-4266642f39ca-kube-api-access-f86sc\") pod \"payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2\" (UID: \"49f36379-dc5e-434c-84be-4266642f39ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 06:27:25.267891 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.267863 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f36379-dc5e-434c-84be-4266642f39ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2\" (UID: \"49f36379-dc5e-434c-84be-4266642f39ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 06:27:25.369504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.369420 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f36379-dc5e-434c-84be-4266642f39ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2\" (UID: \"49f36379-dc5e-434c-84be-4266642f39ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 06:27:25.369504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.369478 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f86sc\" (UniqueName: \"kubernetes.io/projected/49f36379-dc5e-434c-84be-4266642f39ca-kube-api-access-f86sc\") pod \"payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2\" (UID: \"49f36379-dc5e-434c-84be-4266642f39ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 06:27:25.370140 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.370116 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f36379-dc5e-434c-84be-4266642f39ca-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2\" (UID: \"49f36379-dc5e-434c-84be-4266642f39ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 06:27:25.424139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.424116 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f86sc\" (UniqueName: \"kubernetes.io/projected/49f36379-dc5e-434c-84be-4266642f39ca-kube-api-access-f86sc\") pod \"payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2\" (UID: \"49f36379-dc5e-434c-84be-4266642f39ca\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 06:27:25.453913 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.453888 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 06:27:25.820874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:25.820845 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2"] Nov 02 06:27:25.822631 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:27:25.822604 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49f36379_dc5e_434c_84be_4266642f39ca.slice/crio-8bec395897724b04f6e1075596ab5def701b106745eb80accf886c57f63d4819 WatchSource:0}: Error finding container 8bec395897724b04f6e1075596ab5def701b106745eb80accf886c57f63d4819: Status 404 returned error can't find the container with id 8bec395897724b04f6e1075596ab5def701b106745eb80accf886c57f63d4819 Nov 02 06:27:26.099902 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:26.099805 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" event={"ID":"49f36379-dc5e-434c-84be-4266642f39ca","Type":"ContainerStarted","Data":"34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b"} Nov 02 06:27:26.099902 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:26.099867 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" event={"ID":"49f36379-dc5e-434c-84be-4266642f39ca","Type":"ContainerStarted","Data":"8bec395897724b04f6e1075596ab5def701b106745eb80accf886c57f63d4819"} Nov 02 06:27:26.100140 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:26.099968 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 06:27:26.211192 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:26.211144 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" podStartSLOduration=1.211130875 podStartE2EDuration="1.211130875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:26.209258398 +0000 UTC m=+622.657635182" watchObservedRunningTime="2026-11-02 06:27:26.211130875 +0000 UTC m=+622.659507630" Nov 02 06:27:27.061962 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.061932 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz"] Nov 02 06:27:27.065563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.065540 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.163658 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.163623 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-680b7b-serving-cert\"" Nov 02 06:27:27.165330 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.165308 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz"] Nov 02 06:27:27.188181 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.188152 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mczh8\" (UniqueName: \"kubernetes.io/projected/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-kube-api-access-mczh8\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.188181 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.188184 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-metrics-certs\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.188359 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.188214 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.188359 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.188350 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.289916 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.289885 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mczh8\" (UniqueName: \"kubernetes.io/projected/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-kube-api-access-mczh8\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.289916 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.289916 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-metrics-certs\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.290151 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.289956 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.290151 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.290005 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.290151 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:27:27.290127 5025 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-serving-cert: secret "maas-api-e2e-worker-w4-680b7b-serving-cert" not found Nov 02 06:27:27.290309 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:27:27.290205 5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-maas-api-tls podName:0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed nodeName:}" failed. No retries permitted until 2026-11-02 06:27:27.790181908 +0000 UTC m=+624.238558651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-maas-api-tls") pod "maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" (UID: "0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed") : secret "maas-api-e2e-worker-w4-680b7b-serving-cert" not found Nov 02 06:27:27.290793 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.290768 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.292484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.292465 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-metrics-certs\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.319577 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.319347 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:27.345327 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.345301 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mczh8\" (UniqueName: \"kubernetes.io/projected/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-kube-api-access-mczh8\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.508995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.508036 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:27.698575 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.698544 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:27.795543 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.795510 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.797857 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.797830 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-maas-api-tls\") pod \"maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:27.978276 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:27.978194 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:28.341515 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:28.341490 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz"] Nov 02 06:27:28.343731 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:27:28.343701 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcd46fc_2776_4cbb_bb98_fee6a2e1e5ed.slice/crio-6cf85a8851ce85d80ece160053fc15c3925bc0d4994241824ddb91586318e74d WatchSource:0}: Error finding container 6cf85a8851ce85d80ece160053fc15c3925bc0d4994241824ddb91586318e74d: Status 404 returned error can't find the container with id 6cf85a8851ce85d80ece160053fc15c3925bc0d4994241824ddb91586318e74d Nov 02 06:27:29.090422 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:29.090387 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:29.113757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:29.113727 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" event={"ID":"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed","Type":"ContainerStarted","Data":"6cf85a8851ce85d80ece160053fc15c3925bc0d4994241824ddb91586318e74d"} Nov 02 06:27:29.267827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:29.267795 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:30.120286 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:30.120248 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" event={"ID":"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed","Type":"ContainerStarted","Data":"35351843a68a9702168ec92dd91b836f3d1ea380b428b8270098afde1630454a"} Nov 02 06:27:30.120730 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:30.120401 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:30.232527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:30.232469 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" podStartSLOduration=2.406996614 podStartE2EDuration="3.232454241s" totalImagesPullingTime="825.457627ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:30.231283628 +0000 UTC m=+626.679660403" watchObservedRunningTime="2026-11-02 06:27:30.232454241 +0000 UTC m=+626.680830999" Nov 02 06:27:31.105212 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:31.105180 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 06:27:31.402676 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:31.402584 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:31.610101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:31.609271 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:32.518178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:32.518127 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:32.730595 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:27:32.730558 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:27:33.108301 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:33.108267 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 06:27:33.108487 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:33.108353 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 06:27:33.740364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:33.740329 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:33.922239 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:33.922183 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:34.275523 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:34.275479 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:36.046554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:36.046523 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:36.143338 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:36.143309 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:27:36.311180 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:36.305278 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:37.108037 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.108005 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 06:27:37.208900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.208862 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf"] Nov 02 06:27:37.212622 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.212601 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.220061 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.220032 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf"] Nov 02 06:27:37.265246 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.265215 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-cdfce9-serving-cert\"" Nov 02 06:27:37.323066 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.323027 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89b59af-18df-49fe-90c6-3cd91786547c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.323233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.323162 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkdfs\" (UniqueName: \"kubernetes.io/projected/b89b59af-18df-49fe-90c6-3cd91786547c-kube-api-access-nkdfs\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.323304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.323240 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.323304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.323274 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-metrics-certs\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.404138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.404097 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:37.424899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.424863 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkdfs\" (UniqueName: \"kubernetes.io/projected/b89b59af-18df-49fe-90c6-3cd91786547c-kube-api-access-nkdfs\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.425049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.424928 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.425049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.424958 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-metrics-certs\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.425170 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.425142 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89b59af-18df-49fe-90c6-3cd91786547c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.425902 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.425875 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89b59af-18df-49fe-90c6-3cd91786547c-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.427509 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.427481 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-metrics-certs\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.427685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.427668 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-maas-api-tls\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.500729 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.500636 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkdfs\" (UniqueName: \"kubernetes.io/projected/b89b59af-18df-49fe-90c6-3cd91786547c-kube-api-access-nkdfs\") pod \"maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.527246 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.527211 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:37.579593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.579557 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:37.756038 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.755962 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:37.906521 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:37.906005 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf"] Nov 02 06:27:37.907776 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:27:37.907747 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89b59af_18df_49fe_90c6_3cd91786547c.slice/crio-b4b5d792d2bb793327cde3f9966a09ca88e719b8afb4bc17226234a99552a985 WatchSource:0}: Error finding container b4b5d792d2bb793327cde3f9966a09ca88e719b8afb4bc17226234a99552a985: Status 404 returned error can't find the container with id b4b5d792d2bb793327cde3f9966a09ca88e719b8afb4bc17226234a99552a985 Nov 02 06:27:38.175413 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:38.175378 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" event={"ID":"b89b59af-18df-49fe-90c6-3cd91786547c","Type":"ContainerStarted","Data":"b4b5d792d2bb793327cde3f9966a09ca88e719b8afb4bc17226234a99552a985"} Nov 02 06:27:39.181550 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:39.181518 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" event={"ID":"b89b59af-18df-49fe-90c6-3cd91786547c","Type":"ContainerStarted","Data":"ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5"} Nov 02 06:27:39.181940 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:39.181668 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:39.293171 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:39.293101 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" podStartSLOduration=1.468814605 podStartE2EDuration="2.293068216s" totalImagesPullingTime="824.253611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:39.292878517 +0000 UTC m=+635.741255303" watchObservedRunningTime="2026-11-02 06:27:39.293068216 +0000 UTC m=+635.741444975" Nov 02 06:27:40.188093 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:40.188046 5025 generic.go:356] "Generic (PLEG): container finished" podID="60924270-4daa-44e7-a026-cc44a28d77eb" containerID="75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf" exitCode=0 Nov 02 06:27:40.188562 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:40.188124 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" event={"ID":"60924270-4daa-44e7-a026-cc44a28d77eb","Type":"ContainerDied","Data":"75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf"} Nov 02 06:27:40.188728 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:40.188713 5025 scope.go:122] "RemoveContainer" containerID="75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf" Nov 02 06:27:40.189888 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:40.189856 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:40.189991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:40.189923 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:40.189991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:40.189955 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:27:41.212403 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:41.212368 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" event={"ID":"60924270-4daa-44e7-a026-cc44a28d77eb","Type":"ContainerStarted","Data":"8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da"} Nov 02 06:27:42.122563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:42.122529 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:42.123684 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:42.123662 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:27:42.123738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:42.123710 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 06:27:42.462044 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:42.462010 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:42.713539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:42.713413 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:42.910640 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:42.910603 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:43.110118 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:43.109991 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:43.123231 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:43.123206 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:27:43.123374 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:43.123258 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 06:27:44.032914 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:44.032827 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:44.122973 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:44.122934 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:27:44.123173 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:44.122986 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 06:27:45.122731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:45.122700 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:27:45.123116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:45.122750 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 06:27:45.211653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:45.211622 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 06:27:46.122977 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:46.122941 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" start-of-body= Nov 02 06:27:46.123378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:46.123012 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.36:15021/healthz/ready\": dial tcp 10.133.0.36:15021: connect: connection refused" Nov 02 06:27:47.127296 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:47.127264 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:47.128209 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:47.128190 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:27:47.371324 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:47.371289 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:51.582765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.582725 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz"] Nov 02 06:27:51.585765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.585743 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.608087 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.608055 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a402842-33d1-4046-9c1c-bc819bf91032-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.608191 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.608101 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-metrics-certs\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.608191 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.608178 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-maas-api-tls\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.608267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.608206 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65l7q\" (UniqueName: \"kubernetes.io/projected/8a402842-33d1-4046-9c1c-bc819bf91032-kube-api-access-65l7q\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.637517 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.637496 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-bdcd80-serving-cert\"" Nov 02 06:27:51.686483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.686457 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz"] Nov 02 06:27:51.709065 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.709034 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a402842-33d1-4046-9c1c-bc819bf91032-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.709065 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.709070 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-metrics-certs\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.709287 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.709156 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-maas-api-tls\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.709287 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.709197 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65l7q\" (UniqueName: \"kubernetes.io/projected/8a402842-33d1-4046-9c1c-bc819bf91032-kube-api-access-65l7q\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.709390 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:27:51.709290 5025 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-serving-cert: secret "maas-api-e2e-worker-w0-bdcd80-serving-cert" not found Nov 02 06:27:51.709390 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:27:51.709362 5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-maas-api-tls podName:8a402842-33d1-4046-9c1c-bc819bf91032 nodeName:}" failed. No retries permitted until 2026-11-02 06:27:52.209347244 +0000 UTC m=+648.657723998 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-maas-api-tls") pod "maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" (UID: "8a402842-33d1-4046-9c1c-bc819bf91032") : secret "maas-api-e2e-worker-w0-bdcd80-serving-cert" not found Nov 02 06:27:51.709857 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.709832 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a402842-33d1-4046-9c1c-bc819bf91032-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.711786 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.711759 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-metrics-certs\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.763830 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.763798 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65l7q\" (UniqueName: \"kubernetes.io/projected/8a402842-33d1-4046-9c1c-bc819bf91032-kube-api-access-65l7q\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:51.831013 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:51.830975 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:52.023671 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:52.023638 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:52.212960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:52.212919 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-maas-api-tls\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:52.215362 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:52.215339 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-maas-api-tls\") pod \"maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:52.217281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:52.217258 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:52.497238 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:52.497206 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:52.859060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:52.859039 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz"] Nov 02 06:27:52.860755 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:27:52.860725 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a402842_33d1_4046_9c1c_bc819bf91032.slice/crio-73be65d43f06f2eeb6366fe749f720903b82c8232b93742dd9a1cc9a5ad345d4 WatchSource:0}: Error finding container 73be65d43f06f2eeb6366fe749f720903b82c8232b93742dd9a1cc9a5ad345d4: Status 404 returned error can't find the container with id 73be65d43f06f2eeb6366fe749f720903b82c8232b93742dd9a1cc9a5ad345d4 Nov 02 06:27:53.279003 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:53.278971 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" event={"ID":"8a402842-33d1-4046-9c1c-bc819bf91032","Type":"ContainerStarted","Data":"73be65d43f06f2eeb6366fe749f720903b82c8232b93742dd9a1cc9a5ad345d4"} Nov 02 06:27:54.329713 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:54.329673 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" event={"ID":"8a402842-33d1-4046-9c1c-bc819bf91032","Type":"ContainerStarted","Data":"b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625"} Nov 02 06:27:54.330154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:54.329965 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:27:54.463399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:54.463342 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" podStartSLOduration=2.598154466 podStartE2EDuration="3.463329428s" totalImagesPullingTime="865.174962ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:27:54.463283035 +0000 UTC m=+650.911659810" watchObservedRunningTime="2026-11-02 06:27:54.463329428 +0000 UTC m=+650.911706181" Nov 02 06:27:56.358891 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:56.358859 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:56.771972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:56.771750 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:56.977864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:56.977828 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:57.169610 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:57.169578 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:27:59.886634 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:27:59.886530 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:00.340564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:00.340528 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:28:01.079225 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:01.079189 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:28:01.079618 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:01.079252 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:28:01.575720 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:01.575685 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:02.706247 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:28:02.706215 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:28:03.375060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:03.375027 5025 generic.go:356] "Generic (PLEG): container finished" podID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerID="be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8" exitCode=0 Nov 02 06:28:03.375291 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:03.375099 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" event={"ID":"6d9c4a77-9cc7-4478-966e-e0762e9c5c97","Type":"ContainerDied","Data":"be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8"} Nov 02 06:28:03.375484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:03.375468 5025 scope.go:122] "RemoveContainer" containerID="be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8" Nov 02 06:28:03.376723 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:03.376689 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:28:03.376921 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:03.376761 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:28:03.376921 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:03.376798 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:28:04.381454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:04.381419 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" event={"ID":"6d9c4a77-9cc7-4478-966e-e0762e9c5c97","Type":"ContainerStarted","Data":"1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe"} Nov 02 06:28:05.020552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:05.020513 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:28:05.020735 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:05.020712 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 06:28:05.020780 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:05.020763 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 06:28:06.023545 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:06.023510 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 06:28:06.023982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:06.023589 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 06:28:06.023982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:06.023733 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:06.220123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:06.213276 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:06.409377 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:06.409282 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:07.020786 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:07.020747 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 06:28:07.020962 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:07.020802 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 06:28:07.628449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:07.628395 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:08.021103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:08.021045 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 06:28:08.021302 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:08.021132 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 06:28:08.218186 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:08.218154 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:08.410796 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:08.410700 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:09.020623 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:09.020583 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Nov 02 06:28:09.021019 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:09.020643 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Nov 02 06:28:10.024023 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:10.023990 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:28:10.024416 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:10.024399 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:28:10.024964 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:10.024948 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:28:11.009111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:11.009040 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:15.918880 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:15.918845 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:17.188114 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:17.184763 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:20.587244 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:20.587211 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:21.937961 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:21.937919 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:22.528087 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:22.528029 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:22.798858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:22.798778 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:22.994309 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:22.994279 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:25.511595 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:25.511534 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:26.199173 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:26.199140 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:28.286737 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:28.286671 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:28.874567 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:28.874532 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:29.062795 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:29.062761 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:30.466946 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:30.466909 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:32.706735 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:28:32.706693 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:28:35.049608 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:35.049563 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:36.659405 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:36.659328 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:38.654569 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:38.654533 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:39.299757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:39.299714 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:39.486710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:39.486667 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:40.079580 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:40.079543 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:43.804095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:43.804016 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:44.381574 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:44.381542 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:44.564379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:44.564346 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:45.020620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:45.020576 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:49.578136 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:49.578067 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:53.156445 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:53.156402 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:54.498193 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:54.497976 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:54.704239 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:54.704201 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:55.069964 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:55.069932 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:55.256524 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:55.256492 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:59.059698 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:59.059663 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:28:59.637778 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:28:59.637738 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:02.706622 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:29:02.706588 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:29:03.658437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:03.658371 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:07.386883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:07.385936 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:08.262938 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:08.262900 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:12.865678 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:12.865588 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:17.490874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:17.490777 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:17.689516 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:17.689485 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:18.364319 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:18.364275 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:22.069848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:22.069801 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:26.670105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:26.670044 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:31.278416 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:31.278380 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:32.662679 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:32.662643 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:32.707521 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:29:32.707485 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:29:35.870034 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:35.869993 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:38.899912 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:38.899878 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4"] Nov 02 06:29:38.903495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:38.903478 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:38.921989 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:38.921955 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24vhf\" (UniqueName: \"kubernetes.io/projected/90419b03-68d9-4a02-845b-b600b2bb10f7-kube-api-access-24vhf\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:38.922156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:38.922035 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-home\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:38.922156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:38.922135 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90419b03-68d9-4a02-845b-b600b2bb10f7-tls-certs\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:38.922309 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:38.922199 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-dshm\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:38.922309 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:38.922230 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-tmp-dir\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:38.922309 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:38.922291 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-model-cache\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.001205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.001173 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"kube-root-ca.crt\"" Nov 02 06:29:39.001205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.001180 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"openshift-service-ca.crt\"" Nov 02 06:29:39.001447 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.001298 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"premium-w4-680b7b-kserve-self-signed-certs\"" Nov 02 06:29:39.001511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.001487 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"default-dockercfg-8rnrh\"" Nov 02 06:29:39.003058 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.003032 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4"] Nov 02 06:29:39.023464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.023434 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-model-cache\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.023605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.023498 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24vhf\" (UniqueName: \"kubernetes.io/projected/90419b03-68d9-4a02-845b-b600b2bb10f7-kube-api-access-24vhf\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.023605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.023538 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-home\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.023605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.023571 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90419b03-68d9-4a02-845b-b600b2bb10f7-tls-certs\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.023757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.023607 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-dshm\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.023757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.023638 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-tmp-dir\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.023865 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.023829 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-model-cache\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.023956 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.023926 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-home\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.023991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.023976 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-tmp-dir\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.026012 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.025985 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-dshm\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.026241 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.026223 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90419b03-68d9-4a02-845b-b600b2bb10f7-tls-certs\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.041317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.041233 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:39.078642 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.078613 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24vhf\" (UniqueName: \"kubernetes.io/projected/90419b03-68d9-4a02-845b-b600b2bb10f7-kube-api-access-24vhf\") pod \"premium-w4-680b7b-kserve-8668b4f54f-2jxw4\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.218238 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.218202 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:39.597505 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.597472 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4"] Nov 02 06:29:39.598150 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:29:39.598116 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90419b03_68d9_4a02_845b_b600b2bb10f7.slice/crio-d2074fdf565f8cfb4720c201ce25445c7186e3d9b002abb7640f3ba24c3f666e WatchSource:0}: Error finding container d2074fdf565f8cfb4720c201ce25445c7186e3d9b002abb7640f3ba24c3f666e: Status 404 returned error can't find the container with id d2074fdf565f8cfb4720c201ce25445c7186e3d9b002abb7640f3ba24c3f666e Nov 02 06:29:39.890473 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.890392 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" event={"ID":"90419b03-68d9-4a02-845b-b600b2bb10f7","Type":"ContainerStarted","Data":"48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11"} Nov 02 06:29:39.890473 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.890437 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" event={"ID":"90419b03-68d9-4a02-845b-b600b2bb10f7","Type":"ContainerStarted","Data":"d2074fdf565f8cfb4720c201ce25445c7186e3d9b002abb7640f3ba24c3f666e"} Nov 02 06:29:40.000049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:39.999995 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podStartSLOduration=1.999981049 podStartE2EDuration="1.999981049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:29:39.99895399 +0000 UTC m=+756.447330764" watchObservedRunningTime="2026-11-02 06:29:39.999981049 +0000 UTC m=+756.448357805" Nov 02 06:29:40.218559 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:40.218528 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:40.463740 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:40.463581 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:45.072674 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:45.072639 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:49.218491 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:49.218452 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:49.231494 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:49.231465 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:49.697779 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:49.697731 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:49.946996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:49.946967 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:29:52.290142 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:52.290103 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:54.245721 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:54.245684 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:29:58.779491 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:29:58.779393 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:00.186137 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:00.186098 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92"] Nov 02 06:30:00.190038 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:00.190014 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" Nov 02 06:30:00.239503 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:00.239482 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:30:00.341357 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:00.341330 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29944a26-c6cc-422e-9f01-16104c8ec903-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92\" (UID: \"29944a26-c6cc-422e-9f01-16104c8ec903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" Nov 02 06:30:00.341518 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:00.341404 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7bwt\" (UniqueName: \"kubernetes.io/projected/29944a26-c6cc-422e-9f01-16104c8ec903-kube-api-access-b7bwt\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92\" (UID: \"29944a26-c6cc-422e-9f01-16104c8ec903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" Nov 02 06:30:00.372790 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:00.372762 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92"] Nov 02 06:30:00.442598 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:00.442527 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29944a26-c6cc-422e-9f01-16104c8ec903-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92\" (UID: \"29944a26-c6cc-422e-9f01-16104c8ec903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" Nov 02 06:30:00.442725 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:00.442632 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7bwt\" (UniqueName: \"kubernetes.io/projected/29944a26-c6cc-422e-9f01-16104c8ec903-kube-api-access-b7bwt\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92\" (UID: \"29944a26-c6cc-422e-9f01-16104c8ec903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" Nov 02 06:30:00.443334 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:00.443316 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29944a26-c6cc-422e-9f01-16104c8ec903-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92\" (UID: \"29944a26-c6cc-422e-9f01-16104c8ec903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" Nov 02 06:30:00.504784 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:00.504754 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7bwt\" (UniqueName: \"kubernetes.io/projected/29944a26-c6cc-422e-9f01-16104c8ec903-kube-api-access-b7bwt\") pod \"maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92\" (UID: \"29944a26-c6cc-422e-9f01-16104c8ec903\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" Nov 02 06:30:00.804383 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:00.804299 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" Nov 02 06:30:01.212631 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:01.212592 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92"] Nov 02 06:30:01.214727 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:30:01.214694 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29944a26_c6cc_422e_9f01_16104c8ec903.slice/crio-21241b6176e97a1f0fe59ca5f376e37d5695ef9c20670d7fe324c6ee55cedee7 WatchSource:0}: Error finding container 21241b6176e97a1f0fe59ca5f376e37d5695ef9c20670d7fe324c6ee55cedee7: Status 404 returned error can't find the container with id 21241b6176e97a1f0fe59ca5f376e37d5695ef9c20670d7fe324c6ee55cedee7 Nov 02 06:30:01.993879 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:01.993845 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" event={"ID":"29944a26-c6cc-422e-9f01-16104c8ec903","Type":"ContainerStarted","Data":"21241b6176e97a1f0fe59ca5f376e37d5695ef9c20670d7fe324c6ee55cedee7"} Nov 02 06:30:02.707755 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:30:02.707720 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:30:03.375288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:03.375224 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:08.091756 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:08.091714 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:09.029522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:09.029488 5025 generic.go:356] "Generic (PLEG): container finished" podID="29944a26-c6cc-422e-9f01-16104c8ec903" containerID="e99d1a7fdda72dbfd590fe683ee39ba2c7961ef3874e55db88f2c60c7bda2a4b" exitCode=60 Nov 02 06:30:09.029726 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:09.029579 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" event={"ID":"29944a26-c6cc-422e-9f01-16104c8ec903","Type":"ContainerDied","Data":"e99d1a7fdda72dbfd590fe683ee39ba2c7961ef3874e55db88f2c60c7bda2a4b"} Nov 02 06:30:09.029799 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:09.029785 5025 scope.go:122] "RemoveContainer" containerID="e99d1a7fdda72dbfd590fe683ee39ba2c7961ef3874e55db88f2c60c7bda2a4b" Nov 02 06:30:09.868644 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:09.868558 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:10.035882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:10.035851 5025 generic.go:356] "Generic (PLEG): container finished" podID="29944a26-c6cc-422e-9f01-16104c8ec903" containerID="d419832a11054e5d308f9b5c75c04f43a8911903c42a3213399ec94eb346bee0" exitCode=60 Nov 02 06:30:10.036049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:10.035918 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" event={"ID":"29944a26-c6cc-422e-9f01-16104c8ec903","Type":"ContainerDied","Data":"d419832a11054e5d308f9b5c75c04f43a8911903c42a3213399ec94eb346bee0"} Nov 02 06:30:10.036049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:10.035989 5025 scope.go:122] "RemoveContainer" containerID="e99d1a7fdda72dbfd590fe683ee39ba2c7961ef3874e55db88f2c60c7bda2a4b" Nov 02 06:30:10.036257 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:10.036240 5025 scope.go:122] "RemoveContainer" containerID="d419832a11054e5d308f9b5c75c04f43a8911903c42a3213399ec94eb346bee0" Nov 02 06:30:10.036543 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:30:10.036515 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92_odh-ai-gateway-infra(29944a26-c6cc-422e-9f01-16104c8ec903)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" Nov 02 06:30:11.043068 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:11.043040 5025 scope.go:122] "RemoveContainer" containerID="d419832a11054e5d308f9b5c75c04f43a8911903c42a3213399ec94eb346bee0" Nov 02 06:30:11.043465 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:30:11.043304 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92_odh-ai-gateway-infra(29944a26-c6cc-422e-9f01-16104c8ec903)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" Nov 02 06:30:12.685711 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:12.685674 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:17.275425 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:17.275388 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:17.864439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:17.864406 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:20.661959 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:20.661930 5025 scope.go:122] "RemoveContainer" containerID="d419832a11054e5d308f9b5c75c04f43a8911903c42a3213399ec94eb346bee0" Nov 02 06:30:21.090420 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:21.090392 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" event={"ID":"29944a26-c6cc-422e-9f01-16104c8ec903","Type":"ContainerStarted","Data":"f6317b84124f1165ef29e75d4b6f6dfd0af5df041ec77046d1964f6bf90fccd0"} Nov 02 06:30:21.200532 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:21.200483 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" podStartSLOduration=14.600463196 podStartE2EDuration="21.200467422s" totalImagesPullingTime="6.600004226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:30:21.198587492 +0000 UTC m=+797.646964266" watchObservedRunningTime="2026-11-02 06:30:21.200467422 +0000 UTC m=+797.648844179" Nov 02 06:30:21.887760 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:21.887601 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:22.097593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:22.097559 5025 generic.go:356] "Generic (PLEG): container finished" podID="29944a26-c6cc-422e-9f01-16104c8ec903" containerID="f6317b84124f1165ef29e75d4b6f6dfd0af5df041ec77046d1964f6bf90fccd0" exitCode=60 Nov 02 06:30:22.097758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:22.097632 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" event={"ID":"29944a26-c6cc-422e-9f01-16104c8ec903","Type":"ContainerDied","Data":"f6317b84124f1165ef29e75d4b6f6dfd0af5df041ec77046d1964f6bf90fccd0"} Nov 02 06:30:22.097758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:22.097692 5025 scope.go:122] "RemoveContainer" containerID="d419832a11054e5d308f9b5c75c04f43a8911903c42a3213399ec94eb346bee0" Nov 02 06:30:22.097998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:22.097981 5025 scope.go:122] "RemoveContainer" containerID="f6317b84124f1165ef29e75d4b6f6dfd0af5df041ec77046d1964f6bf90fccd0" Nov 02 06:30:22.098272 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:30:22.098253 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92_odh-ai-gateway-infra(29944a26-c6cc-422e-9f01-16104c8ec903)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" Nov 02 06:30:22.220527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:22.220495 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92"] Nov 02 06:30:23.261670 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:23.261648 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" Nov 02 06:30:23.288382 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:23.288340 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29944a26-c6cc-422e-9f01-16104c8ec903-kube-api-access-b7bwt\" (UniqueName: \"kubernetes.io/projected/29944a26-c6cc-422e-9f01-16104c8ec903-kube-api-access-b7bwt\") pod \"29944a26-c6cc-422e-9f01-16104c8ec903\" (UID: \"29944a26-c6cc-422e-9f01-16104c8ec903\") " Nov 02 06:30:23.288523 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:23.288411 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29944a26-c6cc-422e-9f01-16104c8ec903-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29944a26-c6cc-422e-9f01-16104c8ec903-maas-api-service-ca\") pod \"29944a26-c6cc-422e-9f01-16104c8ec903\" (UID: \"29944a26-c6cc-422e-9f01-16104c8ec903\") " Nov 02 06:30:23.288895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:23.288861 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29944a26-c6cc-422e-9f01-16104c8ec903-maas-api-service-ca" pod "29944a26-c6cc-422e-9f01-16104c8ec903" (UID: "29944a26-c6cc-422e-9f01-16104c8ec903"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:30:23.291105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:23.291057 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29944a26-c6cc-422e-9f01-16104c8ec903-kube-api-access-b7bwt" pod "29944a26-c6cc-422e-9f01-16104c8ec903" (UID: "29944a26-c6cc-422e-9f01-16104c8ec903"). InnerVolumeSpecName "kube-api-access-b7bwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:30:23.389817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:23.389744 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7bwt\" (UniqueName: \"kubernetes.io/projected/29944a26-c6cc-422e-9f01-16104c8ec903-kube-api-access-b7bwt\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:23.389817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:23.389770 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29944a26-c6cc-422e-9f01-16104c8ec903-maas-api-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:30:24.117489 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:24.117409 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" event={"ID":"29944a26-c6cc-422e-9f01-16104c8ec903","Type":"ContainerDied","Data":"21241b6176e97a1f0fe59ca5f376e37d5695ef9c20670d7fe324c6ee55cedee7"} Nov 02 06:30:24.117489 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:24.117464 5025 scope.go:122] "RemoveContainer" containerID="f6317b84124f1165ef29e75d4b6f6dfd0af5df041ec77046d1964f6bf90fccd0" Nov 02 06:30:24.117707 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:24.117425 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92" Nov 02 06:30:24.276559 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:24.276510 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92"] Nov 02 06:30:24.281752 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:24.281724 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fb1398-29893350-csh92"] Nov 02 06:30:25.668034 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:25.667992 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" path="/var/lib/kubelet/pods/29944a26-c6cc-422e-9f01-16104c8ec903/volumes" Nov 02 06:30:26.512480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:26.510909 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:31.088905 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:31.088869 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:32.708408 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:30:32.708368 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:30:35.722519 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:35.722467 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:36.435168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:36.435046 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:40.266323 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:40.266279 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:44.874466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:44.874257 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:47.930428 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:47.930398 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w"] Nov 02 06:30:47.930831 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:47.930819 5025 cpu_manager.go:414] "containerMap: removing container" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" containerName="cleanup" Nov 02 06:30:47.930874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:47.930832 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" containerName="cleanup" Nov 02 06:30:47.930874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:47.930841 5025 cpu_manager.go:414] "containerMap: removing container" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" containerName="cleanup" Nov 02 06:30:47.930874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:47.930846 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" containerName="cleanup" Nov 02 06:30:47.930968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:47.930936 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" containerName="cleanup" Nov 02 06:30:47.930968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:47.930943 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" containerName="cleanup" Nov 02 06:30:47.933758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:47.933741 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.006032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.006002 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w"] Nov 02 06:30:48.033166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.033139 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"openshift-service-ca.crt\"" Nov 02 06:30:48.033166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.033145 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"kube-root-ca.crt\"" Nov 02 06:30:48.033166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.033170 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"default-dockercfg-mf5zh\"" Nov 02 06:30:48.033380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.033212 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fb1398\"/\"premium-w3-fb1398-kserve-self-signed-certs\"" Nov 02 06:30:48.048588 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.048562 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-home\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.048694 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.048597 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76zgv\" (UniqueName: \"kubernetes.io/projected/e0264a41-8731-418b-ada7-bf8f950f2e22-kube-api-access-76zgv\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.048694 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.048621 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-model-cache\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.048761 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.048710 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-tmp-dir\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.048761 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.048736 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-dshm\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.048825 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.048783 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0264a41-8731-418b-ada7-bf8f950f2e22-tls-certs\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.085379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.085343 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:48.149611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.149571 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-dshm\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.149798 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.149671 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0264a41-8731-418b-ada7-bf8f950f2e22-tls-certs\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.149798 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.149731 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-home\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.149798 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.149768 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76zgv\" (UniqueName: \"kubernetes.io/projected/e0264a41-8731-418b-ada7-bf8f950f2e22-kube-api-access-76zgv\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.149973 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.149798 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-model-cache\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.149973 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.149854 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-tmp-dir\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.150185 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.150158 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-home\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.150263 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.150191 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-model-cache\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.150339 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.150281 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-tmp-dir\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.151880 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.151860 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-dshm\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.152151 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.152136 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0264a41-8731-418b-ada7-bf8f950f2e22-tls-certs\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.205575 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.205511 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76zgv\" (UniqueName: \"kubernetes.io/projected/e0264a41-8731-418b-ada7-bf8f950f2e22-kube-api-access-76zgv\") pod \"premium-w3-fb1398-kserve-64f46b7c96-m9q9w\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.244954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.244925 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:48.647011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:48.646971 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w"] Nov 02 06:30:49.238005 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:49.237968 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" event={"ID":"e0264a41-8731-418b-ada7-bf8f950f2e22","Type":"ContainerStarted","Data":"eae6ccf54650e0e3071749b7aa98e01f29b8230365b348476ee3131f6916fcc4"} Nov 02 06:30:49.238401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:49.238015 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" event={"ID":"e0264a41-8731-418b-ada7-bf8f950f2e22","Type":"ContainerStarted","Data":"31e8686d30fcf252a1371953565da01b4a7847fd4c71a1482934761cec5f696d"} Nov 02 06:30:49.245290 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:49.245258 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:49.348890 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:49.348840 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podStartSLOduration=2.348826925 podStartE2EDuration="2.348826925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:30:49.34772672 +0000 UTC m=+825.796103495" watchObservedRunningTime="2026-11-02 06:30:49.348826925 +0000 UTC m=+825.797203733" Nov 02 06:30:49.474880 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:49.474848 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:54.185843 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:54.185775 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:30:58.245514 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:58.245480 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:58.258834 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:58.258800 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:58.294747 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:58.294723 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:30:58.736739 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:30:58.736698 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:01.699945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:01.699911 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:02.706975 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:31:02.706930 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:31:03.363395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:03.363348 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:04.710700 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:04.710666 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:04.889462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:04.889432 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:07.996113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:07.992261 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:12.707370 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:12.707336 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:17.277553 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:17.277513 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:18.187910 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:18.187876 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:21.966900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:21.966850 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:26.581613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:26.581570 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:29.342352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.342319 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn"] Nov 02 06:31:29.342749 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.342737 5025 cpu_manager.go:414] "containerMap: removing container" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" containerName="cleanup" Nov 02 06:31:29.342749 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.342750 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" containerName="cleanup" Nov 02 06:31:29.342839 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.342830 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="29944a26-c6cc-422e-9f01-16104c8ec903" containerName="cleanup" Nov 02 06:31:29.346070 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.346052 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.445110 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.445055 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn"] Nov 02 06:31:29.470530 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.470501 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.470692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.470562 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.470692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.470651 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.470692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.470682 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.470799 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.470713 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.470832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.470809 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.470902 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.470887 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5vm\" (UniqueName: \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-kube-api-access-qd5vm\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.470939 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.470916 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.470971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.470964 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.482391 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.482366 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fb1398-openshift-default-dockercfg-rn84l\"" Nov 02 06:31:29.572285 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572256 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.572473 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572294 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.572473 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572324 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.572473 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572374 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.572473 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572404 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.572724 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572551 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.572724 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572633 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.572724 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572680 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.572884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572737 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5vm\" (UniqueName: \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-kube-api-access-qd5vm\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.572884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572809 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.572884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572873 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.573031 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.572959 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.573153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.573133 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.573368 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.573345 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.575085 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.575053 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.575396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.575374 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.629307 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.629238 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5vm\" (UniqueName: \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-kube-api-access-qd5vm\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.629307 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.629257 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:29.658611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:29.658587 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:30.071103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:30.071062 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn"] Nov 02 06:31:30.072105 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:31:30.072062 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67dabe99_aa17_4b0d_b582_ef6fbbc826f1.slice/crio-1a2d16e062b20eaf25fa241393ceacbce42a15d00951712c19c69a89eb87a010 WatchSource:0}: Error finding container 1a2d16e062b20eaf25fa241393ceacbce42a15d00951712c19c69a89eb87a010: Status 404 returned error can't find the container with id 1a2d16e062b20eaf25fa241393ceacbce42a15d00951712c19c69a89eb87a010 Nov 02 06:31:30.074482 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:30.074449 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:31:30.074565 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:30.074518 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:31:30.074565 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:30.074544 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:31:30.439670 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:30.439629 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" event={"ID":"67dabe99-aa17-4b0d-b582-ef6fbbc826f1","Type":"ContainerStarted","Data":"2528049797f3ae3042222247fbd3a3aadbf183d95a86f12cb7e3c4f3af5bb617"} Nov 02 06:31:30.440103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:30.439699 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" event={"ID":"67dabe99-aa17-4b0d-b582-ef6fbbc826f1","Type":"ContainerStarted","Data":"1a2d16e062b20eaf25fa241393ceacbce42a15d00951712c19c69a89eb87a010"} Nov 02 06:31:30.553099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:30.552442 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" podStartSLOduration=1.552424741 podStartE2EDuration="1.552424741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:31:30.550154774 +0000 UTC m=+866.998531530" watchObservedRunningTime="2026-11-02 06:31:30.552424741 +0000 UTC m=+867.000801496" Nov 02 06:31:30.659408 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:30.659376 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:31.179208 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:31.179170 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:31.660865 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:31.660784 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 06:31:31.660865 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:31.660840 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" podUID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 06:31:32.659397 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:32.659364 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 06:31:32.659576 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:32.659427 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" podUID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 06:31:32.732762 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:31:32.732720 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:31:33.659939 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:33.659901 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 06:31:33.660165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:33.659972 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" podUID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 06:31:34.659277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:34.659243 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 06:31:34.659655 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:34.659304 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" podUID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 06:31:35.659921 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:35.659882 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Nov 02 06:31:35.660439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:35.659939 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" podUID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Nov 02 06:31:35.732344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:35.732304 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:37.536931 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:37.536885 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:37.537464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:37.537440 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:37.538404 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:37.538359 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:31:40.371986 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:40.371947 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:45.015958 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.015912 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:45.020894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.020871 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp"] Nov 02 06:31:45.024734 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.024718 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.078335 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.078296 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-680b7b-openshift-default-dockercfg-sccx9\"" Nov 02 06:31:45.127280 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.127248 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp"] Nov 02 06:31:45.166352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.166325 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6vpp\" (UniqueName: \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-kube-api-access-f6vpp\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.166499 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.166365 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.166499 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.166456 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.166499 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.166494 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.166599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.166516 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9765890-d47a-4002-a17a-a4e3afb2efe3-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.166599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.166539 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.166698 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.166650 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.166698 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.166683 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.166763 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.166719 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.267480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.267389 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.267480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.267453 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.267710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.267489 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.267710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.267554 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6vpp\" (UniqueName: \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-kube-api-access-f6vpp\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.267710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.267589 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.267710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.267650 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.267928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.267711 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.267928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.267747 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9765890-d47a-4002-a17a-a4e3afb2efe3-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.267928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.267785 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.267928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.267878 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.268244 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.268218 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.268318 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.268234 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.268318 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.268285 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.268794 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.268772 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9765890-d47a-4002-a17a-a4e3afb2efe3-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.269941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.269916 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.270258 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.270238 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.322259 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.322227 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.322685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.322667 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6vpp\" (UniqueName: \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-kube-api-access-f6vpp\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.338811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.338784 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:45.750121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.750094 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp"] Nov 02 06:31:45.751708 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:31:45.751664 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9765890_d47a_4002_a17a_a4e3afb2efe3.slice/crio-a9dd3f7931d9d03e6ff15f5da9a427a28f1ffefe5268b23fe0f121ba38a853e1 WatchSource:0}: Error finding container a9dd3f7931d9d03e6ff15f5da9a427a28f1ffefe5268b23fe0f121ba38a853e1: Status 404 returned error can't find the container with id a9dd3f7931d9d03e6ff15f5da9a427a28f1ffefe5268b23fe0f121ba38a853e1 Nov 02 06:31:45.754398 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.754361 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:31:45.754518 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.754445 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:31:45.754518 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:45.754493 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:31:46.521944 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:46.521905 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" event={"ID":"a9765890-d47a-4002-a17a-a4e3afb2efe3","Type":"ContainerStarted","Data":"e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31"} Nov 02 06:31:46.522352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:46.521963 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" event={"ID":"a9765890-d47a-4002-a17a-a4e3afb2efe3","Type":"ContainerStarted","Data":"a9dd3f7931d9d03e6ff15f5da9a427a28f1ffefe5268b23fe0f121ba38a853e1"} Nov 02 06:31:46.634617 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:46.634567 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" podStartSLOduration=2.634553043 podStartE2EDuration="2.634553043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:31:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:31:46.633351158 +0000 UTC m=+883.081727930" watchObservedRunningTime="2026-11-02 06:31:46.634553043 +0000 UTC m=+883.082929809" Nov 02 06:31:47.339025 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:47.338985 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:47.340574 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:47.340548 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 06:31:47.340718 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:47.340606 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 06:31:48.339416 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:48.339375 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 06:31:48.339800 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:48.339433 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 06:31:49.339502 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:49.339465 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 06:31:49.340015 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:49.339554 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 06:31:49.634267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:49.634155 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:50.339418 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:50.339384 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 06:31:50.339622 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:50.339452 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 06:31:51.113143 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:51.113091 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:51.339354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:51.339320 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 06:31:51.339521 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:51.339402 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 06:31:52.339532 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:52.339499 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 02 06:31:52.339930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:52.339566 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 02 06:31:53.344184 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:53.344150 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:53.344649 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:53.344557 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:53.345208 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:53.345188 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:31:54.279729 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:54.279690 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:31:58.989882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:31:58.986160 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:02.706920 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:32:02.706886 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:32:03.590921 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:03.590890 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:08.161875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:08.161837 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:12.402719 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:12.402683 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:12.784048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:12.784011 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:17.378319 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:17.378160 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:19.547390 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:19.547360 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:22.185598 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:22.185557 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:22.727569 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:22.727537 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:26.808100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:26.805919 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:31.470717 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:31.470677 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:32.708181 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:32:32.708144 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:32:36.064235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:36.064181 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:40.683124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:40.683094 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:44.092235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.092201 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2"] Nov 02 06:32:44.096016 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.095995 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.147379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.147354 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-cdfce9-openshift-default-dockercfg-j4bz4\"" Nov 02 06:32:44.148803 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.148776 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.148910 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.148817 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.148910 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.148846 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz5v4\" (UniqueName: \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-kube-api-access-zz5v4\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.149050 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.149031 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.149098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.149065 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.149174 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.149155 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.149210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.149192 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.149253 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.149217 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.149291 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.149273 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.197630 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.197595 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2"] Nov 02 06:32:44.250643 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.250614 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.250823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.250653 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.250823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.250679 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.250823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.250719 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.250823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.250758 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.251040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.250835 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.251040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.250883 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5v4\" (UniqueName: \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-kube-api-access-zz5v4\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.251040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.251018 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.251205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.251059 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.251262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.251243 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.251333 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.251312 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.251448 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.251421 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.251511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.251451 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.252149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.252125 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.252918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.252900 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.253441 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.253424 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.304820 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.304792 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.304997 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.304974 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5v4\" (UniqueName: \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-kube-api-access-zz5v4\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.408557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.408478 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:44.819741 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.819712 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2"] Nov 02 06:32:44.821257 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:32:44.821231 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ececdd4_fb2d_41d4_b0e1_1d924a9c21e9.slice/crio-3e20df19d4c5f018c1cb3aa3945e6adee218ef2b8ae97d5715c80a33caea1f8d WatchSource:0}: Error finding container 3e20df19d4c5f018c1cb3aa3945e6adee218ef2b8ae97d5715c80a33caea1f8d: Status 404 returned error can't find the container with id 3e20df19d4c5f018c1cb3aa3945e6adee218ef2b8ae97d5715c80a33caea1f8d Nov 02 06:32:44.823395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.823363 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:32:44.823457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.823447 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:32:44.823493 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.823479 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:32:44.856972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:44.856932 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" event={"ID":"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9","Type":"ContainerStarted","Data":"3e20df19d4c5f018c1cb3aa3945e6adee218ef2b8ae97d5715c80a33caea1f8d"} Nov 02 06:32:45.284267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:45.284066 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:45.863808 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:45.863770 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" event={"ID":"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9","Type":"ContainerStarted","Data":"bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987"} Nov 02 06:32:46.026937 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:46.026877 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" podStartSLOduration=2.026859338 podStartE2EDuration="2.026859338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:32:46.022918679 +0000 UTC m=+942.471295459" watchObservedRunningTime="2026-11-02 06:32:46.026859338 +0000 UTC m=+942.475236095" Nov 02 06:32:46.416366 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:46.416334 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:46.417843 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:46.417816 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 06:32:46.418041 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:46.418016 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" podUID="3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 06:32:47.415731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:47.415697 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 06:32:47.415961 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:47.415811 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" podUID="3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 06:32:48.408938 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:48.408901 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 06:32:48.409597 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:48.408954 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" podUID="3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 06:32:48.750124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:48.750070 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b"] Nov 02 06:32:48.750309 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:48.750287 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" podUID="d713b5c0-e038-4884-979d-d248b3546c50" containerName="istio-proxy" containerID="cri-o://4a936b9100e13437c02297ed3b91ede21a693fac81930ff4046488870d53d65b" gracePeriod=30 Nov 02 06:32:49.409618 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:49.409570 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 06:32:49.410109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:49.409641 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" podUID="3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 06:32:49.996101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:49.990582 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:50.409606 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:50.409515 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Nov 02 06:32:50.409606 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:50.409582 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" podUID="3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Nov 02 06:32:51.674689 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:51.674659 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:51.675275 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:51.674992 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:51.675998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:51.675976 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:32:53.934867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:53.934838 5025 generic.go:356] "Generic (PLEG): container finished" podID="d713b5c0-e038-4884-979d-d248b3546c50" containerID="4a936b9100e13437c02297ed3b91ede21a693fac81930ff4046488870d53d65b" exitCode=0 Nov 02 06:32:53.935248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:53.934940 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" event={"ID":"d713b5c0-e038-4884-979d-d248b3546c50","Type":"ContainerDied","Data":"4a936b9100e13437c02297ed3b91ede21a693fac81930ff4046488870d53d65b"} Nov 02 06:32:54.011409 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.011351 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:32:54.147798 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.147764 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d713b5c0-e038-4884-979d-d248b3546c50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d713b5c0-e038-4884-979d-d248b3546c50-istiod-ca-cert\") pod \"d713b5c0-e038-4884-979d-d248b3546c50\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " Nov 02 06:32:54.147988 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.147817 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d713b5c0-e038-4884-979d-d248b3546c50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d713b5c0-e038-4884-979d-d248b3546c50-istio-podinfo\") pod \"d713b5c0-e038-4884-979d-d248b3546c50\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " Nov 02 06:32:54.147988 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.147846 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-istio-token\" (UniqueName: \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-istio-token\") pod \"d713b5c0-e038-4884-979d-d248b3546c50\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " Nov 02 06:32:54.147988 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.147907 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-kube-api-access-7kjsx\" (UniqueName: \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-kube-api-access-7kjsx\") pod \"d713b5c0-e038-4884-979d-d248b3546c50\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " Nov 02 06:32:54.148201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148010 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-certs\") pod \"d713b5c0-e038-4884-979d-d248b3546c50\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " Nov 02 06:32:54.148201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148115 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-envoy\") pod \"d713b5c0-e038-4884-979d-d248b3546c50\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " Nov 02 06:32:54.148201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148176 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-socket\") pod \"d713b5c0-e038-4884-979d-d248b3546c50\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " Nov 02 06:32:54.148357 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148208 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-data\") pod \"d713b5c0-e038-4884-979d-d248b3546c50\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " Nov 02 06:32:54.148357 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148252 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-credential-socket\") pod \"d713b5c0-e038-4884-979d-d248b3546c50\" (UID: \"d713b5c0-e038-4884-979d-d248b3546c50\") " Nov 02 06:32:54.148357 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148256 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d713b5c0-e038-4884-979d-d248b3546c50-istiod-ca-cert" pod "d713b5c0-e038-4884-979d-d248b3546c50" (UID: "d713b5c0-e038-4884-979d-d248b3546c50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:32:54.148521 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148321 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-certs" pod "d713b5c0-e038-4884-979d-d248b3546c50" (UID: "d713b5c0-e038-4884-979d-d248b3546c50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:32:54.148584 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148544 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-socket" pod "d713b5c0-e038-4884-979d-d248b3546c50" (UID: "d713b5c0-e038-4884-979d-d248b3546c50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:32:54.148642 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148583 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-data" pod "d713b5c0-e038-4884-979d-d248b3546c50" (UID: "d713b5c0-e038-4884-979d-d248b3546c50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:32:54.148804 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148786 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:54.148900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148887 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:54.149001 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148980 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:54.149001 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148999 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d713b5c0-e038-4884-979d-d248b3546c50-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:54.149230 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.148893 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-credential-socket" pod "d713b5c0-e038-4884-979d-d248b3546c50" (UID: "d713b5c0-e038-4884-979d-d248b3546c50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:32:54.150497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.150473 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-kube-api-access-7kjsx" pod "d713b5c0-e038-4884-979d-d248b3546c50" (UID: "d713b5c0-e038-4884-979d-d248b3546c50"). InnerVolumeSpecName "kube-api-access-7kjsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:54.150585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.150525 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-istio-token" pod "d713b5c0-e038-4884-979d-d248b3546c50" (UID: "d713b5c0-e038-4884-979d-d248b3546c50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:32:54.150585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.150557 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d713b5c0-e038-4884-979d-d248b3546c50-istio-podinfo" pod "d713b5c0-e038-4884-979d-d248b3546c50" (UID: "d713b5c0-e038-4884-979d-d248b3546c50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:32:54.150867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.150847 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-envoy" pod "d713b5c0-e038-4884-979d-d248b3546c50" (UID: "d713b5c0-e038-4884-979d-d248b3546c50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:32:54.250322 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.250291 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:54.250322 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.250318 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d713b5c0-e038-4884-979d-d248b3546c50-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:54.250322 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.250329 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:54.250563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.250338 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kjsx\" (UniqueName: \"kubernetes.io/projected/d713b5c0-e038-4884-979d-d248b3546c50-kube-api-access-7kjsx\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:54.250563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.250347 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d713b5c0-e038-4884-979d-d248b3546c50-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:32:54.591118 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.587280 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:32:54.941844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.941816 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" Nov 02 06:32:54.941844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.941826 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b" event={"ID":"d713b5c0-e038-4884-979d-d248b3546c50","Type":"ContainerDied","Data":"2281810f2b7c09eeb77f128f5778ae428cdc1d63fab702c140de94f9a8985fb8"} Nov 02 06:32:54.942392 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:54.941891 5025 scope.go:122] "RemoveContainer" containerID="4a936b9100e13437c02297ed3b91ede21a693fac81930ff4046488870d53d65b" Nov 02 06:32:55.146629 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:55.146601 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b"] Nov 02 06:32:55.193850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:55.193788 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-m478b"] Nov 02 06:32:55.668010 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:55.667971 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d713b5c0-e038-4884-979d-d248b3546c50" path="/var/lib/kubelet/pods/d713b5c0-e038-4884-979d-d248b3546c50/volumes" Nov 02 06:32:59.196305 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:32:59.196269 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:00.844113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.844069 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86"] Nov 02 06:33:00.844566 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.844553 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d713b5c0-e038-4884-979d-d248b3546c50" containerName="istio-proxy" Nov 02 06:33:00.844613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.844567 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d713b5c0-e038-4884-979d-d248b3546c50" containerName="istio-proxy" Nov 02 06:33:00.844660 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.844651 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d713b5c0-e038-4884-979d-d248b3546c50" containerName="istio-proxy" Nov 02 06:33:00.849251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.849227 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:00.899009 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.898977 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-6ff85e-openshift-default-dockercfg-f2wx5\"" Nov 02 06:33:00.923071 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.923034 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:00.923225 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.923150 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:00.923281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.923230 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:00.923281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.923258 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:00.923386 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.923364 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:00.923437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.923402 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c654f68-3867-449f-8f59-626b947927b4-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:00.923512 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.923496 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d7ll\" (UniqueName: \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-kube-api-access-7d7ll\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:00.923558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.923535 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c654f68-3867-449f-8f59-626b947927b4-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:00.923684 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.923669 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:00.948361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:00.948329 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86"] Nov 02 06:33:01.024971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.024933 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.025175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.024977 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.025175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.025004 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.025175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.025068 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.025175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.025114 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.025175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.025145 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.025175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.025170 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c654f68-3867-449f-8f59-626b947927b4-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.025485 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.025232 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d7ll\" (UniqueName: \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-kube-api-access-7d7ll\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.025485 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.025267 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c654f68-3867-449f-8f59-626b947927b4-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.025485 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.025398 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.025885 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.025856 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.025994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.025951 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.026061 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.026001 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.026225 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.026196 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c654f68-3867-449f-8f59-626b947927b4-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.027425 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.027401 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.028645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.028624 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c654f68-3867-449f-8f59-626b947927b4-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.080826 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.080792 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.080968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.080874 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d7ll\" (UniqueName: \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-kube-api-access-7d7ll\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.163555 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.163533 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:01.589381 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.589354 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86"] Nov 02 06:33:01.590700 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:33:01.590670 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c654f68_3867_449f_8f59_626b947927b4.slice/crio-4e48a379bc2ca9d6666b8908537d19493b25286510510f50e1b5e124fc13bfd4 WatchSource:0}: Error finding container 4e48a379bc2ca9d6666b8908537d19493b25286510510f50e1b5e124fc13bfd4: Status 404 returned error can't find the container with id 4e48a379bc2ca9d6666b8908537d19493b25286510510f50e1b5e124fc13bfd4 Nov 02 06:33:01.593410 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.593377 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:33:01.593493 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.593447 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:33:01.593493 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.593477 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:33:01.993562 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.993521 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" event={"ID":"7c654f68-3867-449f-8f59-626b947927b4","Type":"ContainerStarted","Data":"a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e"} Nov 02 06:33:01.994018 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:01.993609 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" event={"ID":"7c654f68-3867-449f-8f59-626b947927b4","Type":"ContainerStarted","Data":"4e48a379bc2ca9d6666b8908537d19493b25286510510f50e1b5e124fc13bfd4"} Nov 02 06:33:02.168119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:02.164008 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:02.168119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:02.166830 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 06:33:02.168119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:02.166909 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" podUID="7c654f68-3867-449f-8f59-626b947927b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 06:33:02.206102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:02.205190 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" podStartSLOduration=2.202689312 podStartE2EDuration="2.202689312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:33:02.152920734 +0000 UTC m=+958.601297521" watchObservedRunningTime="2026-11-02 06:33:02.202689312 +0000 UTC m=+958.651066069" Nov 02 06:33:02.759637 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:33:02.759600 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:33:03.166095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:03.164469 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 06:33:03.166095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:03.164541 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" podUID="7c654f68-3867-449f-8f59-626b947927b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 06:33:03.842146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:03.842110 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:04.163970 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:04.163936 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 06:33:04.164145 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:04.164009 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" podUID="7c654f68-3867-449f-8f59-626b947927b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 06:33:05.164968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:05.164928 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 06:33:05.165400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:05.164988 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" podUID="7c654f68-3867-449f-8f59-626b947927b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 06:33:06.164087 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:06.164049 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 06:33:06.164261 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:06.164126 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" podUID="7c654f68-3867-449f-8f59-626b947927b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 06:33:07.163924 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:07.163891 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 06:33:07.164372 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:07.163955 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" podUID="7c654f68-3867-449f-8f59-626b947927b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 06:33:08.164560 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:08.164524 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Nov 02 06:33:08.164965 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:08.164605 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" podUID="7c654f68-3867-449f-8f59-626b947927b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Nov 02 06:33:08.576768 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:08.576678 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:09.168461 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:09.168430 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:09.168930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:09.168900 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:09.169605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:09.169582 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:33:11.187329 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:11.187292 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:13.180027 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:13.179923 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:16.108308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:16.108264 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:17.773144 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:17.773100 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:18.862210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:18.862173 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:21.105359 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.105329 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp"] Nov 02 06:33:21.108932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.108915 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.193387 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.193358 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-home\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.193541 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.193411 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-dshm\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.193590 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.193565 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-model-cache\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.193627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.193600 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flz9h\" (UniqueName: \"kubernetes.io/projected/8ac70ab7-07cd-4046-8882-fbf50037c5ba-kube-api-access-flz9h\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.193706 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.193695 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tls-certs\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.193751 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.193732 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tmp-dir\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.205773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.205747 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-680b7b\"/\"distinct-w4-680b7b-kserve-self-signed-certs\"" Nov 02 06:33:21.207774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.207756 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp"] Nov 02 06:33:21.276583 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.276550 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:21.295100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.295033 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-model-cache\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.295100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.295100 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flz9h\" (UniqueName: \"kubernetes.io/projected/8ac70ab7-07cd-4046-8882-fbf50037c5ba-kube-api-access-flz9h\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.295330 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.295161 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tls-certs\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.295330 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.295214 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tmp-dir\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.295330 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.295284 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-home\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.295458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.295357 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-dshm\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.296154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.296120 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-model-cache\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.296449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.296417 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-home\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.296545 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.296505 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tmp-dir\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.298023 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.298005 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-dshm\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.298327 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.298164 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tls-certs\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.350990 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.350963 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flz9h\" (UniqueName: \"kubernetes.io/projected/8ac70ab7-07cd-4046-8882-fbf50037c5ba-kube-api-access-flz9h\") pod \"distinct-w4-680b7b-kserve-6958688496-lgpqp\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.420730 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.420690 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:21.821409 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:21.821378 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp"] Nov 02 06:33:21.822881 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:33:21.822856 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ac70ab7_07cd_4046_8882_fbf50037c5ba.slice/crio-a1cfb101f45432daefed22caee2c2870404e0140d27a014fdfe12d8594a6a34d WatchSource:0}: Error finding container a1cfb101f45432daefed22caee2c2870404e0140d27a014fdfe12d8594a6a34d: Status 404 returned error can't find the container with id a1cfb101f45432daefed22caee2c2870404e0140d27a014fdfe12d8594a6a34d Nov 02 06:33:22.145465 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:22.145381 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" event={"ID":"8ac70ab7-07cd-4046-8882-fbf50037c5ba","Type":"ContainerStarted","Data":"d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f"} Nov 02 06:33:22.145465 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:22.145425 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" event={"ID":"8ac70ab7-07cd-4046-8882-fbf50037c5ba","Type":"ContainerStarted","Data":"a1cfb101f45432daefed22caee2c2870404e0140d27a014fdfe12d8594a6a34d"} Nov 02 06:33:22.255568 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:22.255519 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podStartSLOduration=1.25550517 podStartE2EDuration="1.25550517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:33:22.253619963 +0000 UTC m=+978.701996723" watchObservedRunningTime="2026-11-02 06:33:22.25550517 +0000 UTC m=+978.703881937" Nov 02 06:33:22.415469 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:22.415430 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:22.420842 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:22.420809 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:26.982100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:26.981242 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:29.696248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:29.695624 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:31.421706 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:31.421671 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:31.434271 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:31.434243 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:31.629512 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:31.629471 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:32.207821 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:32.207795 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:33:32.360789 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:32.360739 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:32.707306 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:33:32.707273 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:33:36.189944 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:36.189910 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:40.771509 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:40.771295 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:44.964545 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:44.964460 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9"] Nov 02 06:33:44.968631 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:44.968613 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.018769 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.018741 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-bdcd80-openshift-default-dockercfg-h4jqn\"" Nov 02 06:33:45.070909 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.070880 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9"] Nov 02 06:33:45.077324 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.077295 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.077454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.077336 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.077496 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.077452 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.077496 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.077485 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.077572 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.077521 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad1a864e-1183-4c56-a21a-c76b3130d948-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.077614 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.077578 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvvhc\" (UniqueName: \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-kube-api-access-pvvhc\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.077657 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.077619 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.077657 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.077640 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.077793 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.077766 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.178880 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.178846 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.178889 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.178937 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.178956 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.178982 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad1a864e-1183-4c56-a21a-c76b3130d948-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.179018 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvvhc\" (UniqueName: \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-kube-api-access-pvvhc\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.179098 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.179131 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.179148 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179521 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.179416 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179521 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.179500 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.179517 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179771 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.179750 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.179834 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.179817 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad1a864e-1183-4c56-a21a-c76b3130d948-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.181329 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.181309 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.181519 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.181503 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.233423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.233353 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.233528 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.233454 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvvhc\" (UniqueName: \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-kube-api-access-pvvhc\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.281757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.281715 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:45.381096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.379489 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:45.694736 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.694711 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9"] Nov 02 06:33:45.695838 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:33:45.695811 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1a864e_1183_4c56_a21a_c76b3130d948.slice/crio-2aac1f175b618049bc54b3f1be88110a48a5b80bf0a741c77e8e65ccb84bb24f WatchSource:0}: Error finding container 2aac1f175b618049bc54b3f1be88110a48a5b80bf0a741c77e8e65ccb84bb24f: Status 404 returned error can't find the container with id 2aac1f175b618049bc54b3f1be88110a48a5b80bf0a741c77e8e65ccb84bb24f Nov 02 06:33:45.698395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.698363 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:33:45.698472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.698430 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:33:45.698472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:45.698457 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:33:46.267787 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:46.267750 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" event={"ID":"ad1a864e-1183-4c56-a21a-c76b3130d948","Type":"ContainerStarted","Data":"c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4"} Nov 02 06:33:46.268202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:46.267809 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" event={"ID":"ad1a864e-1183-4c56-a21a-c76b3130d948","Type":"ContainerStarted","Data":"2aac1f175b618049bc54b3f1be88110a48a5b80bf0a741c77e8e65ccb84bb24f"} Nov 02 06:33:46.282654 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:46.282628 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:46.303260 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:46.303216 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 06:33:46.303471 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:46.303284 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 06:33:46.381924 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:46.381870 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" podStartSLOduration=2.381855442 podStartE2EDuration="2.381855442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:33:46.380110359 +0000 UTC m=+1002.828487137" watchObservedRunningTime="2026-11-02 06:33:46.381855442 +0000 UTC m=+1002.830232197" Nov 02 06:33:47.282732 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:47.282701 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 06:33:47.283166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:47.282750 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 06:33:48.282274 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:48.282241 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 06:33:48.282484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:48.282322 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 06:33:49.282226 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:49.282187 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 06:33:49.282644 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:49.282246 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 06:33:50.106695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:50.106623 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:50.282477 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:50.282444 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 06:33:50.282902 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:50.282498 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 06:33:51.282823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:51.282784 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Nov 02 06:33:51.283341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:51.282858 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Nov 02 06:33:52.562037 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:52.562002 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:52.562468 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:52.562443 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:52.563491 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:52.563465 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:33:54.663216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:54.663182 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:59.254480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:59.254443 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:33:59.477024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:33:59.476968 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:02.706427 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:34:02.706393 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:34:03.876096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:03.876047 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:08.489097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:08.486922 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:10.167977 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:10.167945 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:13.108301 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:13.108207 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:14.028353 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:14.028317 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:14.207531 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:14.207497 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:17.672458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:17.672424 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:19.211208 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:19.211174 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:22.280649 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:22.280601 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:26.210995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:26.209719 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:26.893844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:26.890921 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:31.483120 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:31.482089 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:32.706735 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:34:32.706702 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:34:36.069116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:36.068740 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:40.760426 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:40.760368 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:45.428886 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:45.428812 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:46.833183 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.833148 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq"] Nov 02 06:34:46.839417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.839390 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:46.894093 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.894046 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq"] Nov 02 06:34:46.927777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.927743 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-tmp-dir\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:46.927777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.927793 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6d6102-1694-4edc-baec-59156cdcf81e-tls-certs\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:46.928045 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.927897 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxkm2\" (UniqueName: \"kubernetes.io/projected/7c6d6102-1694-4edc-baec-59156cdcf81e-kube-api-access-nxkm2\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:46.928045 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.927930 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-dshm\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:46.928045 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.927961 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-home\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:46.928230 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.928062 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-model-cache\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:46.938860 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.938832 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"kube-root-ca.crt\"" Nov 02 06:34:46.939337 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.939317 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"openshift-service-ca.crt\"" Nov 02 06:34:46.939337 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.939325 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"default-dockercfg-6lvdv\"" Nov 02 06:34:46.939532 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.939397 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-bdcd80\"/\"distinct-2-w0-bdcd80-kserve-self-signed-certs\"" Nov 02 06:34:46.967884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:46.967814 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:47.029063 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.029029 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-tmp-dir\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.029251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.029102 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6d6102-1694-4edc-baec-59156cdcf81e-tls-certs\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.029251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.029152 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkm2\" (UniqueName: \"kubernetes.io/projected/7c6d6102-1694-4edc-baec-59156cdcf81e-kube-api-access-nxkm2\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.029251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.029173 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-dshm\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.029251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.029202 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-home\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.029251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.029239 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-model-cache\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.029535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.029472 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-tmp-dir\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.029637 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.029617 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-home\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.029679 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.029646 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-model-cache\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.031415 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.031389 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-dshm\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.031647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.031631 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6d6102-1694-4edc-baec-59156cdcf81e-tls-certs\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.088101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.087994 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkm2\" (UniqueName: \"kubernetes.io/projected/7c6d6102-1694-4edc-baec-59156cdcf81e-kube-api-access-nxkm2\") pod \"distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.151845 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.151811 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:47.559404 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.559378 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq"] Nov 02 06:34:47.561537 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:34:47.561507 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c6d6102_1694_4edc_baec_59156cdcf81e.slice/crio-36e11a44180272787d73cf43f28238057a06a5f49eb3c763fcbba24f8265bc2d WatchSource:0}: Error finding container 36e11a44180272787d73cf43f28238057a06a5f49eb3c763fcbba24f8265bc2d: Status 404 returned error can't find the container with id 36e11a44180272787d73cf43f28238057a06a5f49eb3c763fcbba24f8265bc2d Nov 02 06:34:47.642647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.642611 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" event={"ID":"7c6d6102-1694-4edc-baec-59156cdcf81e","Type":"ContainerStarted","Data":"112541316a8a03b0992492c6051cd6959e921a844f7cb2d67f73299eba1d7fcb"} Nov 02 06:34:47.642799 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.642658 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" event={"ID":"7c6d6102-1694-4edc-baec-59156cdcf81e","Type":"ContainerStarted","Data":"36e11a44180272787d73cf43f28238057a06a5f49eb3c763fcbba24f8265bc2d"} Nov 02 06:34:47.754227 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:47.754175 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podStartSLOduration=1.754159027 podStartE2EDuration="1.754159027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:34:47.752811276 +0000 UTC m=+1064.201188063" watchObservedRunningTime="2026-11-02 06:34:47.754159027 +0000 UTC m=+1064.202535816" Nov 02 06:34:48.152039 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:48.152002 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:50.045515 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:50.045480 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:54.561770 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:54.560114 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:34:57.152947 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:57.152914 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:57.165069 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:57.165039 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:57.713912 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:57.713878 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:34:59.105836 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:34:59.105793 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:01.715054 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.715023 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh"] Nov 02 06:35:01.718899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.718883 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.790216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.790183 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.790340 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.790300 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.790340 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.790330 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.790424 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.790395 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zfv9\" (UniqueName: \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-kube-api-access-2zfv9\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.790469 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.790456 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.790561 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.790537 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.790686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.790665 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.790757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.790707 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.790757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.790736 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.817864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.817839 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh"] Nov 02 06:35:01.891870 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.891836 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.891870 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.891877 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.891894 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.891947 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.891986 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.892009 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.892102 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zfv9\" (UniqueName: \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-kube-api-access-2zfv9\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.892138 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892476 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.892178 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892476 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.892376 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892476 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.892432 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.892553 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.892636 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.892754 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.892734 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.894875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.894848 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.895004 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.894989 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.946762 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.946722 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:01.946898 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:01.946778 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zfv9\" (UniqueName: \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-kube-api-access-2zfv9\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:02.033928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:02.033846 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:02.447552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:02.447521 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh"] Nov 02 06:35:02.448661 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:35:02.448630 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9dab3c5_b0ed_4b9c_954f_8350b87248bd.slice/crio-0da516ec749a744b2c113032e003aead1d6bf3ffab67bc5d50d8df916fd49d53 WatchSource:0}: Error finding container 0da516ec749a744b2c113032e003aead1d6bf3ffab67bc5d50d8df916fd49d53: Status 404 returned error can't find the container with id 0da516ec749a744b2c113032e003aead1d6bf3ffab67bc5d50d8df916fd49d53 Nov 02 06:35:02.451101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:02.451050 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:35:02.451188 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:02.451148 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:35:02.451252 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:02.451196 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:35:02.709273 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:35:02.709176 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:35:02.737926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:02.737887 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" event={"ID":"b9dab3c5-b0ed-4b9c-954f-8350b87248bd","Type":"ContainerStarted","Data":"e04e6214d96ab6a7f45291dd9ab5c5db36fef301842a4f4e6fcb69a054fbcf2a"} Nov 02 06:35:02.738308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:02.737957 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" event={"ID":"b9dab3c5-b0ed-4b9c-954f-8350b87248bd","Type":"ContainerStarted","Data":"0da516ec749a744b2c113032e003aead1d6bf3ffab67bc5d50d8df916fd49d53"} Nov 02 06:35:02.848868 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:02.848800 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" podStartSLOduration=1.84878169 podStartE2EDuration="1.84878169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:35:02.847852656 +0000 UTC m=+1079.296229436" watchObservedRunningTime="2026-11-02 06:35:02.84878169 +0000 UTC m=+1079.297158447" Nov 02 06:35:03.035138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:03.035013 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:03.036616 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:03.036571 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 06:35:03.036754 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:03.036637 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 06:35:03.676430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:03.676375 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:04.034462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:04.034382 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 06:35:04.034462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:04.034432 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 06:35:05.034662 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:05.034621 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 06:35:05.035056 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:05.034683 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 06:35:06.034345 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:06.034312 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 06:35:06.034516 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:06.034362 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 06:35:07.035252 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:07.035204 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 06:35:07.035695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:07.035259 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 06:35:08.035101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:08.035040 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 06:35:08.035358 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:08.035120 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 06:35:08.208766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:08.208727 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:09.035024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:09.034981 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Nov 02 06:35:09.035213 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:09.035054 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Nov 02 06:35:10.038724 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:10.038692 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:10.039132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:10.038952 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:10.039605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:10.039585 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:35:12.720098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:12.710699 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:17.266404 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:17.266121 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:19.534332 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:19.534285 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:21.919167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:21.919125 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:24.683647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:24.683609 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:26.553637 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:26.553602 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:31.098723 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:31.098648 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:32.707527 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:35:32.707496 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:35:35.691659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:35.691572 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:39.288391 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:39.288340 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:39.886815 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:39.886773 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:40.373110 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:40.367198 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:45.076996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:45.076935 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:48.065510 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.065476 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2"] Nov 02 06:35:48.069385 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.069368 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.165153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.165109 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-home\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.165325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.165169 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-tmp-dir\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.165325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.165238 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7v8t\" (UniqueName: \"kubernetes.io/projected/158b7fb1-559a-4c01-be74-7359c5633cc0-kube-api-access-h7v8t\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.165325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.165283 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-model-cache\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.165431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.165372 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158b7fb1-559a-4c01-be74-7359c5633cc0-tls-certs\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.165431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.165421 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-dshm\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.168634 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.168611 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"kube-root-ca.crt\"" Nov 02 06:35:48.168763 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.168683 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"default-dockercfg-28q6p\"" Nov 02 06:35:48.168763 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.168731 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"openshift-service-ca.crt\"" Nov 02 06:35:48.168849 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.168764 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-cdfce9\"/\"distinct-2-w1-cdfce9-kserve-self-signed-certs\"" Nov 02 06:35:48.171011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.170991 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2"] Nov 02 06:35:48.217185 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.217138 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:48.267102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.267036 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-home\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.267102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.267086 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-tmp-dir\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.267350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.267118 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7v8t\" (UniqueName: \"kubernetes.io/projected/158b7fb1-559a-4c01-be74-7359c5633cc0-kube-api-access-h7v8t\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.267350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.267144 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-model-cache\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.267350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.267219 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158b7fb1-559a-4c01-be74-7359c5633cc0-tls-certs\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.267350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.267260 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-dshm\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.267903 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.267867 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-model-cache\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.268089 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.268054 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-tmp-dir\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.268237 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.268216 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-home\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.276848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.276821 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158b7fb1-559a-4c01-be74-7359c5633cc0-tls-certs\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.277547 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.277520 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-dshm\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.322861 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.322802 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7v8t\" (UniqueName: \"kubernetes.io/projected/158b7fb1-559a-4c01-be74-7359c5633cc0-kube-api-access-h7v8t\") pod \"distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.381210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.381174 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:48.790655 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.790630 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2"] Nov 02 06:35:48.792344 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:35:48.792319 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod158b7fb1_559a_4c01_be74_7359c5633cc0.slice/crio-6a61f79c00cf92c803d1be89b1278f457062b3cfd7280644c6d02d3e1f8a36c7 WatchSource:0}: Error finding container 6a61f79c00cf92c803d1be89b1278f457062b3cfd7280644c6d02d3e1f8a36c7: Status 404 returned error can't find the container with id 6a61f79c00cf92c803d1be89b1278f457062b3cfd7280644c6d02d3e1f8a36c7 Nov 02 06:35:48.970629 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.970591 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" event={"ID":"158b7fb1-559a-4c01-be74-7359c5633cc0","Type":"ContainerStarted","Data":"5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3"} Nov 02 06:35:48.970801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:48.970640 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" event={"ID":"158b7fb1-559a-4c01-be74-7359c5633cc0","Type":"ContainerStarted","Data":"6a61f79c00cf92c803d1be89b1278f457062b3cfd7280644c6d02d3e1f8a36c7"} Nov 02 06:35:49.081235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:49.081134 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podStartSLOduration=1.081119363 podStartE2EDuration="1.081119363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:35:49.080218564 +0000 UTC m=+1125.528595319" watchObservedRunningTime="2026-11-02 06:35:49.081119363 +0000 UTC m=+1125.529496119" Nov 02 06:35:49.382281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:49.382202 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:49.811227 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:49.811194 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:53.314098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:53.314052 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:54.370997 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:54.370965 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:54.897765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:54.897732 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:57.105056 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:57.105024 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:57.283593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:57.283562 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:58.381669 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:58.381639 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:58.393686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:58.393657 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:35:58.990577 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:58.990542 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:35:59.055716 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:35:59.055686 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:36:01.280657 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.280616 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm"] Nov 02 06:36:01.284360 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.284342 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.370895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.370861 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm"] Nov 02 06:36:01.414522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.414488 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.414522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.414523 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.414735 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.414548 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.414735 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.414696 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.414807 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.414777 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.414840 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.414830 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.414889 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.414845 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.414928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.414899 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v48hf\" (UniqueName: \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-kube-api-access-v48hf\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.414962 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.414948 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.516136 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516070 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.516298 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516253 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.516298 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516285 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.516393 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516322 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.516393 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516360 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.516469 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516405 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.516513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516470 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.516513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516495 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.516642 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516542 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v48hf\" (UniqueName: \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-kube-api-access-v48hf\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.516749 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516719 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.516968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516754 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.517150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516774 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.517150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.516866 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.517282 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.517272 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.518680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.518660 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.518767 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.518722 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.571693 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.571633 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.571810 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.571702 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v48hf\" (UniqueName: \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-kube-api-access-v48hf\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:01.599400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:01.599366 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:02.018058 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:02.018033 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm"] Nov 02 06:36:02.019897 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:36:02.019868 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded9a87c7_20ed_4343_9eb7_2b713d4cd569.slice/crio-8468071a0eb56adbd1a43fd1342b62948bc7cd938392ccef336121ad84da93ba WatchSource:0}: Error finding container 8468071a0eb56adbd1a43fd1342b62948bc7cd938392ccef336121ad84da93ba: Status 404 returned error can't find the container with id 8468071a0eb56adbd1a43fd1342b62948bc7cd938392ccef336121ad84da93ba Nov 02 06:36:02.022462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:02.022429 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:02.022565 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:02.022506 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:02.022565 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:02.022552 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:02.063720 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:02.063689 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" event={"ID":"ed9a87c7-20ed-4343-9eb7-2b713d4cd569","Type":"ContainerStarted","Data":"8468071a0eb56adbd1a43fd1342b62948bc7cd938392ccef336121ad84da93ba"} Nov 02 06:36:02.749706 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:36:02.749668 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:36:03.072047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:03.072008 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" event={"ID":"ed9a87c7-20ed-4343-9eb7-2b713d4cd569","Type":"ContainerStarted","Data":"9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385"} Nov 02 06:36:03.276908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:03.276850 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" podStartSLOduration=2.276836819 podStartE2EDuration="2.276836819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:03.276486375 +0000 UTC m=+1139.724863131" watchObservedRunningTime="2026-11-02 06:36:03.276836819 +0000 UTC m=+1139.725213631" Nov 02 06:36:03.627101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:03.625168 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:03.634116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:03.633916 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 06:36:03.634116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:03.633968 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" podUID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 06:36:03.657001 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:03.656969 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:03.904153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:03.899465 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:04.609242 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:04.609204 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 06:36:04.610115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:04.610044 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" podUID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 06:36:05.608742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:05.608253 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 06:36:05.608742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:05.608317 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" podUID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 06:36:06.601099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:06.600420 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 06:36:06.601099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:06.600480 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" podUID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 06:36:07.603542 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:07.603499 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 06:36:07.604178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:07.604142 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" podUID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 06:36:08.292287 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:08.292237 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:08.599996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:08.599906 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 02 06:36:08.599996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:08.599978 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" podUID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 02 06:36:09.604900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:09.604868 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:09.605417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:09.605124 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:09.605766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:09.605748 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:36:12.904493 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:12.904454 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:16.944697 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:16.944661 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z"] Nov 02 06:36:16.949134 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:16.949104 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.041885 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.041845 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9wzh\" (UniqueName: \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-kube-api-access-v9wzh\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.042105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.042036 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.042105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.042099 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.042213 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.042186 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.042267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.042234 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41dade0-68ae-4d60-99de-33151bf4b2b8-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.042355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.042340 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.042396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.042387 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.042447 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.042409 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.042546 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.042526 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.048304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.048276 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z"] Nov 02 06:36:17.143984 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.143945 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.143984 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.143982 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144240 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144021 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144240 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144063 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9wzh\" (UniqueName: \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-kube-api-access-v9wzh\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144240 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144165 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144240 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144186 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144240 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144222 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144247 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41dade0-68ae-4d60-99de-33151bf4b2b8-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144288 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144387 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-data\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144675 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144518 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144732 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144665 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-certs\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144801 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-credential-socket\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.144929 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.144909 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41dade0-68ae-4d60-99de-33151bf4b2b8-istiod-ca-cert\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.146571 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.146548 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-envoy\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.146767 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.146748 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-podinfo\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.202945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.202663 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-token\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.202945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.202715 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9wzh\" (UniqueName: \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-kube-api-access-v9wzh\") pod \"e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.262738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.262703 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:17.600769 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.594703 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:17.684395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.684371 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z"] Nov 02 06:36:17.685654 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:36:17.685623 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41dade0_68ae_4d60_99de_33151bf4b2b8.slice/crio-57bc1c0155aace7485bdacb05ca37e358d8983ced8026538e9568315556a360d WatchSource:0}: Error finding container 57bc1c0155aace7485bdacb05ca37e358d8983ced8026538e9568315556a360d: Status 404 returned error can't find the container with id 57bc1c0155aace7485bdacb05ca37e358d8983ced8026538e9568315556a360d Nov 02 06:36:17.688067 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.688033 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:17.688180 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.688142 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:17.688244 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:17.688189 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:18.218653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:18.218608 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" event={"ID":"c41dade0-68ae-4d60-99de-33151bf4b2b8","Type":"ContainerStarted","Data":"4bdd630cfe5ca1d893f423e4be03fcddef22e90e69e62044638b0931a2e4d6a7"} Nov 02 06:36:18.219097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:18.218713 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" event={"ID":"c41dade0-68ae-4d60-99de-33151bf4b2b8","Type":"ContainerStarted","Data":"57bc1c0155aace7485bdacb05ca37e358d8983ced8026538e9568315556a360d"} Nov 02 06:36:18.263932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:18.263891 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:18.265297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:18.265271 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 06:36:18.265426 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:18.265318 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 06:36:18.329932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:18.329860 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" podStartSLOduration=2.329843106 podStartE2EDuration="2.329843106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:18.328869847 +0000 UTC m=+1154.777246605" watchObservedRunningTime="2026-11-02 06:36:18.329843106 +0000 UTC m=+1154.778219881" Nov 02 06:36:19.264063 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:19.264025 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 06:36:19.264506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:19.264104 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 06:36:19.906099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:19.903308 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:20.263355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:20.263322 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 06:36:20.263522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:20.263376 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 06:36:21.263611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:21.263578 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 06:36:21.263987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:21.263631 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 06:36:22.263619 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:22.263586 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 06:36:22.264003 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:22.263638 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 06:36:22.315686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:22.315627 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:23.263047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:23.263012 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 06:36:23.263232 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:23.263067 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 06:36:24.263473 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:24.263437 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 06:36:24.264005 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:24.263499 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 06:36:25.267852 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:25.267821 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:25.268367 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:25.268239 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:25.268984 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:25.268966 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:36:26.880430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:26.880369 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:30.743336 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.743305 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2"] Nov 02 06:36:30.749106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.748353 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.832174 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.832137 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.832341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.832179 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4885f2f1-b780-46b0-90f9-91f624acc4f9-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.832341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.832258 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.832341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.832307 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.832449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.832362 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.832449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.832380 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf4vn\" (UniqueName: \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-kube-api-access-pf4vn\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.832564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.832492 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.832600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.832564 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.832600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.832581 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.848512 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.848482 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2"] Nov 02 06:36:30.933623 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.933590 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.933811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.933642 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.933811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.933806 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.933932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.933919 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.934002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.933946 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4885f2f1-b780-46b0-90f9-91f624acc4f9-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.934002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.933953 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.934002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.933975 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.934200 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.934011 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.934200 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.934043 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.934200 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.934098 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf4vn\" (UniqueName: \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-kube-api-access-pf4vn\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.934200 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.934041 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.934406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.934333 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.934447 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.934405 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.934643 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.934617 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4885f2f1-b780-46b0-90f9-91f624acc4f9-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.936205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.936185 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.936500 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.936481 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.988599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.988570 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf4vn\" (UniqueName: \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-kube-api-access-pf4vn\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:30.988599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:30.988591 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:31.064313 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:31.064226 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:31.472662 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:31.472491 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:31.487803 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:31.487695 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2"] Nov 02 06:36:31.510187 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:31.510149 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:31.510289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:31.510223 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:31.510289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:31.510265 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:32.322096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:32.321383 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" event={"ID":"4885f2f1-b780-46b0-90f9-91f624acc4f9","Type":"ContainerStarted","Data":"813d4c91244c821f4102e068ecc8154a2a2c94a5aa9088b06a4a3b8ce52d181f"} Nov 02 06:36:32.322096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:32.321454 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" event={"ID":"4885f2f1-b780-46b0-90f9-91f624acc4f9","Type":"ContainerStarted","Data":"b98bb22b38650daa1c4da03bcadce0e54c45b298db238b8879199255157f6e34"} Nov 02 06:36:32.492097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:32.468068 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" podStartSLOduration=2.468048664 podStartE2EDuration="2.468048664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:32.465870408 +0000 UTC m=+1168.914247164" watchObservedRunningTime="2026-11-02 06:36:32.468048664 +0000 UTC m=+1168.916425421" Nov 02 06:36:32.761579 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:36:32.761519 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:36:33.074168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:33.073117 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:33.078097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:33.074881 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:36:33.078097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:33.074942 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 06:36:33.994103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:33.985208 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:36:34.063513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:34.062175 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 06:36:34.067047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:34.065112 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:36:34.067047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:34.065162 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 06:36:34.222107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:34.210186 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:36:34.288160 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:34.287167 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:36:34.507097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:34.506836 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:36:34.716100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:34.715177 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:36:34.998963 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:34.998851 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:36:35.003904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.003872 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:36:35.075103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.071955 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:36:35.079100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.075834 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:36:35.097222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.097180 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:36:35.097400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.097260 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 06:36:35.240108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.239222 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:36:35.240108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.239652 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:36:35.294319 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.294215 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:36:35.336099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.331794 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:36:35.408101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.382903 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:36:35.484097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.457457 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr"] Nov 02 06:36:35.484097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.457712 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" containerName="istio-proxy" containerID="cri-o://3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c" gracePeriod=30 Nov 02 06:36:35.544107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.542175 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:36:35.544107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.542625 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:36:35.777103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:35.742833 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 06:36:36.020098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:36.012344 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 06:36:36.055316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:36.055214 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:36:36.058553 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:36.058518 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:36:36.060858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:36.060835 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:36:36.084569 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:36.084526 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:36:36.084847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:36.084816 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 06:36:36.147464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:36.137969 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:36:36.166124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:36.165667 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:36:36.196099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:36.176649 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:36:36.322256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:36.322162 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:37.065503 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:37.065470 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:36:37.065903 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:37.065524 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 06:36:38.064630 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:38.064596 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:36:38.064835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:38.064659 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 06:36:39.065376 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:39.065336 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Nov 02 06:36:39.065891 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:39.065400 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Nov 02 06:36:40.071436 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.071396 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:40.072031 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.072008 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:40.073693 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.073669 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:36:40.098433 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.098401 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-zftfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:36:40.098577 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.098463 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:36:40.802668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.802550 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:40.824841 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.824813 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:36:40.897886 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.897857 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-socket\") pod \"ec38b35a-07e4-4a82-9444-727ede78e928\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " Nov 02 06:36:40.898043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.897935 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec38b35a-07e4-4a82-9444-727ede78e928-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec38b35a-07e4-4a82-9444-727ede78e928-istiod-ca-cert\") pod \"ec38b35a-07e4-4a82-9444-727ede78e928\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " Nov 02 06:36:40.898043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898002 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec38b35a-07e4-4a82-9444-727ede78e928-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec38b35a-07e4-4a82-9444-727ede78e928-istio-podinfo\") pod \"ec38b35a-07e4-4a82-9444-727ede78e928\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " Nov 02 06:36:40.898043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898020 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-data\") pod \"ec38b35a-07e4-4a82-9444-727ede78e928\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " Nov 02 06:36:40.898222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898059 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-istio-token\" (UniqueName: \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-istio-token\") pod \"ec38b35a-07e4-4a82-9444-727ede78e928\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " Nov 02 06:36:40.898222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898131 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-kube-api-access-nkjs8\" (UniqueName: \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-kube-api-access-nkjs8\") pod \"ec38b35a-07e4-4a82-9444-727ede78e928\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " Nov 02 06:36:40.898222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898148 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-socket" pod "ec38b35a-07e4-4a82-9444-727ede78e928" (UID: "ec38b35a-07e4-4a82-9444-727ede78e928"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:36:40.898222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898175 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-certs\") pod \"ec38b35a-07e4-4a82-9444-727ede78e928\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " Nov 02 06:36:40.898438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898257 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-credential-socket\") pod \"ec38b35a-07e4-4a82-9444-727ede78e928\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " Nov 02 06:36:40.898438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898298 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-envoy\") pod \"ec38b35a-07e4-4a82-9444-727ede78e928\" (UID: \"ec38b35a-07e4-4a82-9444-727ede78e928\") " Nov 02 06:36:40.898438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898362 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-data" pod "ec38b35a-07e4-4a82-9444-727ede78e928" (UID: "ec38b35a-07e4-4a82-9444-727ede78e928"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:36:40.898438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898397 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec38b35a-07e4-4a82-9444-727ede78e928-istiod-ca-cert" pod "ec38b35a-07e4-4a82-9444-727ede78e928" (UID: "ec38b35a-07e4-4a82-9444-727ede78e928"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:36:40.898788 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898505 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-certs" pod "ec38b35a-07e4-4a82-9444-727ede78e928" (UID: "ec38b35a-07e4-4a82-9444-727ede78e928"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:36:40.898895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898788 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:40.898895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898808 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec38b35a-07e4-4a82-9444-727ede78e928-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:40.898895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898822 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:40.898895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.898571 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-credential-socket" pod "ec38b35a-07e4-4a82-9444-727ede78e928" (UID: "ec38b35a-07e4-4a82-9444-727ede78e928"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:36:40.900306 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.900279 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec38b35a-07e4-4a82-9444-727ede78e928-istio-podinfo" pod "ec38b35a-07e4-4a82-9444-727ede78e928" (UID: "ec38b35a-07e4-4a82-9444-727ede78e928"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:36:40.900420 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.900350 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-istio-token" pod "ec38b35a-07e4-4a82-9444-727ede78e928" (UID: "ec38b35a-07e4-4a82-9444-727ede78e928"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:40.900681 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.900650 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-kube-api-access-nkjs8" pod "ec38b35a-07e4-4a82-9444-727ede78e928" (UID: "ec38b35a-07e4-4a82-9444-727ede78e928"). InnerVolumeSpecName "kube-api-access-nkjs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:36:40.900681 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:40.900667 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-envoy" pod "ec38b35a-07e4-4a82-9444-727ede78e928" (UID: "ec38b35a-07e4-4a82-9444-727ede78e928"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:36:41.000472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.000437 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec38b35a-07e4-4a82-9444-727ede78e928-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:41.000472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.000466 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:41.000472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.000475 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkjs8\" (UniqueName: \"kubernetes.io/projected/ec38b35a-07e4-4a82-9444-727ede78e928-kube-api-access-nkjs8\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:41.000698 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.000487 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:41.000698 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.000496 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:41.000698 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.000504 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec38b35a-07e4-4a82-9444-727ede78e928-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:36:41.451957 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.451922 5025 generic.go:356] "Generic (PLEG): container finished" podID="ec38b35a-07e4-4a82-9444-727ede78e928" containerID="3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c" exitCode=0 Nov 02 06:36:41.452448 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.451968 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" Nov 02 06:36:41.452448 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.452004 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" event={"ID":"ec38b35a-07e4-4a82-9444-727ede78e928","Type":"ContainerDied","Data":"3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c"} Nov 02 06:36:41.452448 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.452059 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr" event={"ID":"ec38b35a-07e4-4a82-9444-727ede78e928","Type":"ContainerDied","Data":"2e43b33712a93b443430fa01ed19b6fee0032c43faab5f772551aad6edd81667"} Nov 02 06:36:41.452448 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.452105 5025 scope.go:122] "RemoveContainer" containerID="3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c" Nov 02 06:36:41.462674 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.462653 5025 scope.go:122] "RemoveContainer" containerID="3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c" Nov 02 06:36:41.462932 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:36:41.462914 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c\": container with ID starting with 3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c not found: ID does not exist" containerID="3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c" Nov 02 06:36:41.462977 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.462940 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c"} err="failed to get container status \"3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c\": rpc error: code = NotFound desc = could not find container \"3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c\": container with ID starting with 3021d7a3ae16628d4113c6a9b01a21520d4920f15ec55179a2d6f0fcc8bb2a0c not found: ID does not exist" Nov 02 06:36:41.655389 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.655352 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr"] Nov 02 06:36:41.710869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:41.703823 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-zftfr"] Nov 02 06:36:43.668803 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:43.668770 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" path="/var/lib/kubelet/pods/ec38b35a-07e4-4a82-9444-727ede78e928/volumes" Nov 02 06:36:45.287619 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:45.287494 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:46.601617 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.601580 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5"] Nov 02 06:36:46.602325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.602304 5025 cpu_manager.go:414] "containerMap: removing container" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" containerName="istio-proxy" Nov 02 06:36:46.602325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.602326 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" containerName="istio-proxy" Nov 02 06:36:46.602496 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.602470 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec38b35a-07e4-4a82-9444-727ede78e928" containerName="istio-proxy" Nov 02 06:36:46.607128 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.607105 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.711852 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.711817 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5"] Nov 02 06:36:46.773850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.773799 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.773850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.773850 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.774065 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.773945 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.774065 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.774056 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ssb4\" (UniqueName: \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-kube-api-access-8ssb4\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.774223 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.774201 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.774283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.774236 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.774452 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.774428 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.774554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.774535 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.774646 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.774628 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.876399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.876310 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.876399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.876395 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.876638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.876447 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.876638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.876510 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.876638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.876535 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.876791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.876648 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.876791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.876706 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ssb4\" (UniqueName: \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-kube-api-access-8ssb4\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.876905 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.876806 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.876905 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.876831 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.876905 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.876872 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.876905 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.876890 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.877146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.877103 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.877189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.877147 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.877584 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.877562 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.879154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.879122 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.879251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.879209 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.930712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.930679 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:46.930835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:46.930808 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ssb4\" (UniqueName: \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-kube-api-access-8ssb4\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:47.219673 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:47.219635 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:47.637450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:47.637363 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5"] Nov 02 06:36:47.669837 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:36:47.669809 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb575a0a5_5c30_4169_bc8b_e2aeb933d15f.slice/crio-00d3b37eaa43f289f8c73af0887f9d6ae9ae75c422bccab32c9079a397af35e6 WatchSource:0}: Error finding container 00d3b37eaa43f289f8c73af0887f9d6ae9ae75c422bccab32c9079a397af35e6: Status 404 returned error can't find the container with id 00d3b37eaa43f289f8c73af0887f9d6ae9ae75c422bccab32c9079a397af35e6 Nov 02 06:36:47.672182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:47.672142 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:47.672285 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:47.672217 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:47.672285 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:47.672248 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:36:48.515645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:48.515606 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" event={"ID":"b575a0a5-5c30-4169-bc8b-e2aeb933d15f","Type":"ContainerStarted","Data":"975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0"} Nov 02 06:36:48.515822 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:48.515664 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" event={"ID":"b575a0a5-5c30-4169-bc8b-e2aeb933d15f","Type":"ContainerStarted","Data":"00d3b37eaa43f289f8c73af0887f9d6ae9ae75c422bccab32c9079a397af35e6"} Nov 02 06:36:48.753410 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:48.753361 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" podStartSLOduration=2.753346573 podStartE2EDuration="2.753346573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:36:48.704625238 +0000 UTC m=+1185.153002019" watchObservedRunningTime="2026-11-02 06:36:48.753346573 +0000 UTC m=+1185.201723328" Nov 02 06:36:49.219895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:49.219861 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:49.221347 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:49.221316 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 06:36:49.221488 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:49.221370 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 06:36:50.220517 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:50.220482 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 06:36:50.221011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:50.220542 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 06:36:50.509085 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:50.508989 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:36:51.220269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:51.220232 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 06:36:51.220454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:51.220287 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 06:36:52.220983 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:52.220945 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 06:36:52.221399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:52.221001 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 06:36:53.220762 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:53.220721 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 06:36:53.220939 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:53.220780 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 06:36:54.220196 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:54.220153 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Nov 02 06:36:54.220570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:54.220226 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Nov 02 06:36:55.225139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:55.225102 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:55.225560 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:55.225387 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:55.226167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:55.226143 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:36:55.514094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:36:55.513977 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:00.298661 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:00.298625 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:02.708118 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:37:02.708067 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:37:04.929556 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:04.929495 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:08.388099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:08.387153 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:08.624100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:08.620154 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:08.624100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:08.620219 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:09.065411 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.065360 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 06:37:09.124266 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.124214 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:09.147242 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.147190 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:09.183102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.182597 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:09.183102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.182964 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 06:37:09.252099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.248682 5025 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:09.388263 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.388157 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:09.390671 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.390643 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:37:09.392010 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.391988 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:09.405270 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.405236 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:09.406284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.406255 5025 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:09.637135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.636844 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:37:09.795672 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:09.795637 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:14.428344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:14.428309 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:19.108103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:19.108012 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:19.925233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:19.925183 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:21.567736 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:21.567695 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:23.703923 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:23.703844 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:28.276465 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:28.276379 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:29.100870 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.100838 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7135-948182"] Nov 02 06:37:29.104527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.104509 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-948182" Nov 02 06:37:29.154842 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.154803 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 06:37:29.155289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.155268 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-948182"] Nov 02 06:37:29.179694 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.179649 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vrp9\" (UniqueName: \"kubernetes.io/projected/849e78b5-270f-408d-a9be-931b9759e29f-kube-api-access-8vrp9\") pod \"test-curl-7135-948182\" (UID: \"849e78b5-270f-408d-a9be-931b9759e29f\") " pod="opendatahub/test-curl-7135-948182" Nov 02 06:37:29.199528 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.199491 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 06:37:29.201775 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.201747 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7g9gq\"" Nov 02 06:37:29.280426 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.280393 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vrp9\" (UniqueName: \"kubernetes.io/projected/849e78b5-270f-408d-a9be-931b9759e29f-kube-api-access-8vrp9\") pod \"test-curl-7135-948182\" (UID: \"849e78b5-270f-408d-a9be-931b9759e29f\") " pod="opendatahub/test-curl-7135-948182" Nov 02 06:37:29.334566 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.334535 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vrp9\" (UniqueName: \"kubernetes.io/projected/849e78b5-270f-408d-a9be-931b9759e29f-kube-api-access-8vrp9\") pod \"test-curl-7135-948182\" (UID: \"849e78b5-270f-408d-a9be-931b9759e29f\") " pod="opendatahub/test-curl-7135-948182" Nov 02 06:37:29.416017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.415987 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-948182" Nov 02 06:37:29.781652 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.781616 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-948182"] Nov 02 06:37:29.782171 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:37:29.782144 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod849e78b5_270f_408d_a9be_931b9759e29f.slice/crio-9f118d6027418a96671f73ecd825c7da5611cc89af58603fa22fab6577741bec WatchSource:0}: Error finding container 9f118d6027418a96671f73ecd825c7da5611cc89af58603fa22fab6577741bec: Status 404 returned error can't find the container with id 9f118d6027418a96671f73ecd825c7da5611cc89af58603fa22fab6577741bec Nov 02 06:37:29.783908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:29.783891 5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:37:30.083800 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:30.083718 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-948182" event={"ID":"849e78b5-270f-408d-a9be-931b9759e29f","Type":"ContainerStarted","Data":"9f118d6027418a96671f73ecd825c7da5611cc89af58603fa22fab6577741bec"} Nov 02 06:37:31.927605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:31.927566 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t"] Nov 02 06:37:31.932186 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:31.932165 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.020267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.020203 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.020449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.020307 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.020449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.020339 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.020449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.020365 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.020600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.020484 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.020600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.020547 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.020600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.020586 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.020709 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.020671 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.020749 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.020718 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjbkv\" (UniqueName: \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-kube-api-access-pjbkv\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.033746 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.033714 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t"] Nov 02 06:37:32.121683 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.121651 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.121869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.121697 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.121869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.121718 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.121869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.121734 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.121869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.121754 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.121869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.121782 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.121869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.121809 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.122228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.121885 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.122228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.121935 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjbkv\" (UniqueName: \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-kube-api-access-pjbkv\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.122228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.122069 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.122228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.122127 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-credential-socket\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.122228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.122177 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-certs\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.122493 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.122222 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-data\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.122812 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.122786 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istiod-ca-cert\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.124375 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.124349 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-podinfo\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.124574 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.124547 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-envoy\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.186567 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.186498 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-token\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.192551 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.192523 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjbkv\" (UniqueName: \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-kube-api-access-pjbkv\") pod \"e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.246381 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.246351 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:32.319666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.319346 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh"] Nov 02 06:37:32.324813 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.324787 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.369980 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.369941 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf"] Nov 02 06:37:32.374991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.374965 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.380966 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.380882 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh"] Nov 02 06:37:32.426429 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.426336 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.428401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.426397 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.428401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.426988 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.428401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.427045 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.428401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.427127 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99czx\" (UniqueName: \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-kube-api-access-99czx\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.428401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.427162 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d090f221-e10d-4266-882a-43e4047e9f33-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.428401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.427215 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d090f221-e10d-4266-882a-43e4047e9f33-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.428401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.427254 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.430563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.430191 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.430934 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.430914 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf"] Nov 02 06:37:32.531003 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.530915 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.531003 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.530966 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.531231 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531025 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkvkg\" (UniqueName: \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-kube-api-access-lkvkg\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.531231 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531071 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.531231 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531126 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.531354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531249 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.531354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531294 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.531354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531339 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.531504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531377 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b39f394-283b-4454-8ec6-485f30315d32-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.531504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531407 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.531504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531433 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.531504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531480 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99czx\" (UniqueName: \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-kube-api-access-99czx\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.531703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531511 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.531703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531526 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d090f221-e10d-4266-882a-43e4047e9f33-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.531703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531569 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.531703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531608 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d090f221-e10d-4266-882a-43e4047e9f33-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.531703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531650 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.531949 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531776 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.531949 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.531912 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b39f394-283b-4454-8ec6-485f30315d32-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.532048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.532016 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.532203 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.532180 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d090f221-e10d-4266-882a-43e4047e9f33-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.532292 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.532275 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.532399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.532378 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.533666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.533646 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.534212 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.534196 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d090f221-e10d-4266-882a-43e4047e9f33-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.587619 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.587584 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99czx\" (UniqueName: \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-kube-api-access-99czx\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.587769 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.587589 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.633822 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.633788 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.633822 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.633855 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b39f394-283b-4454-8ec6-485f30315d32-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.633924 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.633963 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkvkg\" (UniqueName: \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-kube-api-access-lkvkg\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.634004 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.634045 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.634123 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b39f394-283b-4454-8ec6-485f30315d32-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.634154 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634501 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.634193 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634501 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.634223 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634610 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.634569 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.634623 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b39f394-283b-4454-8ec6-485f30315d32-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.634623 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.634864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.634834 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.636831 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.636806 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.637128 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.637109 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b39f394-283b-4454-8ec6-485f30315d32-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.640208 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.640186 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:32.665560 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:37:32.665533 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffc5c9e_1a15_4d4c_9762_18fc84808b8f.slice/crio-b8ea68272b57f4a258e30ab66d0da9c45b1ba6fbb41bfc38dc3ebc032922273f WatchSource:0}: Error finding container b8ea68272b57f4a258e30ab66d0da9c45b1ba6fbb41bfc38dc3ebc032922273f: Status 404 returned error can't find the container with id b8ea68272b57f4a258e30ab66d0da9c45b1ba6fbb41bfc38dc3ebc032922273f Nov 02 06:37:32.666289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.666264 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t"] Nov 02 06:37:32.668328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.668294 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:32.668413 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.668365 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:32.668413 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.668407 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:32.689137 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.689112 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.689496 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.689475 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkvkg\" (UniqueName: \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-kube-api-access-lkvkg\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.693117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:32.692258 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:32.715248 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:37:32.715214 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:37:33.023955 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:37:33.023916 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd090f221_e10d_4266_882a_43e4047e9f33.slice/crio-35fc0d9ed27fc8418b1616f2468959c65a38f7a3de37b1000640a0ff9bc8689b WatchSource:0}: Error finding container 35fc0d9ed27fc8418b1616f2468959c65a38f7a3de37b1000640a0ff9bc8689b: Status 404 returned error can't find the container with id 35fc0d9ed27fc8418b1616f2468959c65a38f7a3de37b1000640a0ff9bc8689b Nov 02 06:37:33.024413 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.024296 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh"] Nov 02 06:37:33.026889 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.026855 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.026983 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.026924 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.026983 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.026954 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.114098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.111097 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:33.153156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.142745 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf"] Nov 02 06:37:33.158846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.158338 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.158846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.158493 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.158846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.158534 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:33.159579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.159381 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" event={"ID":"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f","Type":"ContainerStarted","Data":"299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75"} Nov 02 06:37:33.159579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.159437 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" event={"ID":"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f","Type":"ContainerStarted","Data":"b8ea68272b57f4a258e30ab66d0da9c45b1ba6fbb41bfc38dc3ebc032922273f"} Nov 02 06:37:33.164859 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.164834 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" event={"ID":"d090f221-e10d-4266-882a-43e4047e9f33","Type":"ContainerStarted","Data":"35fc0d9ed27fc8418b1616f2468959c65a38f7a3de37b1000640a0ff9bc8689b"} Nov 02 06:37:33.248034 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.247169 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:33.248186 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.248145 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 06:37:33.248230 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.248194 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 06:37:33.272109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:33.271052 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podStartSLOduration=2.271033601 podStartE2EDuration="2.271033601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:33.270047292 +0000 UTC m=+1229.718424078" watchObservedRunningTime="2026-11-02 06:37:33.271033601 +0000 UTC m=+1229.719410375" Nov 02 06:37:34.175814 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.175765 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" event={"ID":"8b39f394-283b-4454-8ec6-485f30315d32","Type":"ContainerStarted","Data":"d01ba258a7ecc282fba60584316d3119e67870563442c438ff4e98624195ecff"} Nov 02 06:37:34.176310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.175860 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" event={"ID":"8b39f394-283b-4454-8ec6-485f30315d32","Type":"ContainerStarted","Data":"43daa11731926422cb1ffebe0167cc378a5e39e6d997c05ac14df9f9820abc11"} Nov 02 06:37:34.178925 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.178897 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" event={"ID":"d090f221-e10d-4266-882a-43e4047e9f33","Type":"ContainerStarted","Data":"b1dd497e535b3b755e1cfcecbabe5d40aee402e1dfa17fb6d2dab5b41596cad5"} Nov 02 06:37:34.261118 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.258597 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 06:37:34.261118 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.258686 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 06:37:34.294141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.292390 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podStartSLOduration=2.292370744 podStartE2EDuration="2.292370744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:34.290289028 +0000 UTC m=+1230.738665807" watchObservedRunningTime="2026-11-02 06:37:34.292370744 +0000 UTC m=+1230.740747502" Nov 02 06:37:34.665171 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.663413 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:34.669443 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.669403 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:37:34.669620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.669474 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 06:37:34.715102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.702931 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:34.715102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.703058 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 06:37:34.715102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:34.703144 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 06:37:35.248100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:35.247519 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 06:37:35.248100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:35.247581 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 06:37:35.644157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:35.643696 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:37:35.644157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:35.643776 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 06:37:35.695103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:35.694513 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 06:37:35.695103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:35.694582 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 06:37:36.263642 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:36.263598 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 06:37:36.264208 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:36.263661 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 06:37:36.669455 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:36.669418 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:37:36.669648 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:36.669491 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 06:37:36.757236 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:36.744021 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 06:37:36.757236 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:36.744108 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 06:37:37.264341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:37.264109 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 06:37:37.264341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:37.264176 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 06:37:37.270775 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:37.270747 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-948182" event={"ID":"849e78b5-270f-408d-a9be-931b9759e29f","Type":"ContainerStarted","Data":"b2c821b8ed0976c74b40d1bd016375863f13015530597e24e392881b41bb895b"} Nov 02 06:37:37.394100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:37.393405 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podStartSLOduration=5.393381083 podStartE2EDuration="5.393381083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:34.445060932 +0000 UTC m=+1230.893437689" watchObservedRunningTime="2026-11-02 06:37:37.393381083 +0000 UTC m=+1233.841757840" Nov 02 06:37:37.418098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:37.414735 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7135-948182" podStartSLOduration=1.319429373 podStartE2EDuration="8.414715258s" totalImagesPullingTime="7.095285885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:37.39449625 +0000 UTC m=+1233.842873006" watchObservedRunningTime="2026-11-02 06:37:37.414715258 +0000 UTC m=+1233.863092015" Nov 02 06:37:37.650152 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:37.649166 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:37:37.650152 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:37.649228 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 06:37:37.706109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:37.706045 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 06:37:37.706284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:37.706130 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 06:37:38.255100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:38.254496 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 06:37:38.255100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:38.254566 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 06:37:38.388882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:38.388846 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:38.648757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:38.646612 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:37:38.648757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:38.646682 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 06:37:38.750495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:38.708669 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 06:37:38.750495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:38.708732 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 06:37:38.751650 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:38.751472 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:38.954882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:38.954674 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:39.175103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.172205 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7135-948182"] Nov 02 06:37:39.176798 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.176756 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7135-948182"] Nov 02 06:37:39.254264 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.254172 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 06:37:39.254515 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.254484 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 06:37:39.352102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.348486 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7135-948182" podUID="849e78b5-270f-408d-a9be-931b9759e29f" containerName="test-curl-7135-948182" containerID="cri-o://b2c821b8ed0976c74b40d1bd016375863f13015530597e24e392881b41bb895b" gracePeriod=2 Nov 02 06:37:39.352102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.349663 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7135-721487"] Nov 02 06:37:39.352102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.350639 5025 cpu_manager.go:414] "containerMap: removing container" podUID="849e78b5-270f-408d-a9be-931b9759e29f" containerName="test-curl-7135-948182" Nov 02 06:37:39.352102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.350659 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849e78b5-270f-408d-a9be-931b9759e29f" containerName="test-curl-7135-948182" Nov 02 06:37:39.352102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.350851 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="849e78b5-270f-408d-a9be-931b9759e29f" containerName="test-curl-7135-948182" Nov 02 06:37:39.354823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.354802 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:39.373926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.373897 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-721487"] Nov 02 06:37:39.457147 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:37:39.413323 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-948182\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="849e78b5-270f-408d-a9be-931b9759e29f" pod="opendatahub/test-curl-7135-948182" Nov 02 06:37:39.457147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.414668 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klwxc\" (UniqueName: \"kubernetes.io/projected/8f22d5c9-c5b6-4e48-acf1-ab44f69925a5-kube-api-access-klwxc\") pod \"test-curl-7135-721487\" (UID: \"8f22d5c9-c5b6-4e48-acf1-ab44f69925a5\") " pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:39.458562 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.458530 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:39.461707 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:37:39.461526 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-948182\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="849e78b5-270f-408d-a9be-931b9759e29f" pod="opendatahub/test-curl-7135-948182" Nov 02 06:37:39.536163 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.535541 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klwxc\" (UniqueName: \"kubernetes.io/projected/8f22d5c9-c5b6-4e48-acf1-ab44f69925a5-kube-api-access-klwxc\") pod \"test-curl-7135-721487\" (UID: \"8f22d5c9-c5b6-4e48-acf1-ab44f69925a5\") " pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:39.670881 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.670543 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:37:39.670881 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.670607 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 06:37:39.672531 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.672502 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klwxc\" (UniqueName: \"kubernetes.io/projected/8f22d5c9-c5b6-4e48-acf1-ab44f69925a5-kube-api-access-klwxc\") pod \"test-curl-7135-721487\" (UID: \"8f22d5c9-c5b6-4e48-acf1-ab44f69925a5\") " pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:39.673212 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.673189 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:39.712736 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.712689 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 06:37:39.712953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:39.712922 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 06:37:40.247311 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:40.246843 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 02 06:37:40.247311 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:40.246901 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 02 06:37:40.330861 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:40.329163 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-721487"] Nov 02 06:37:40.361060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:40.361028 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-721487" event={"ID":"8f22d5c9-c5b6-4e48-acf1-ab44f69925a5","Type":"ContainerStarted","Data":"bfaa61f05bdf33241873303b4e0c287a3905aa9cf9727341972a76959b937d72"} Nov 02 06:37:40.656437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:40.641067 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:37:40.656437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:40.641138 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 06:37:40.702125 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:40.693523 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 06:37:40.702125 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:40.693597 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 06:37:41.650098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:41.646672 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 02 06:37:41.650098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:41.646736 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 02 06:37:41.733170 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:41.706122 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 02 06:37:41.733170 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:41.706185 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 02 06:37:42.070497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.069279 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 06:37:42.070497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.070125 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded" Nov 02 06:37:42.144010 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.143172 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.173603 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.173559 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.179661 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.179624 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 06:37:42.306103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.304183 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:42.306103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.304242 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.435100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.433393 5025 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/health\": context deadline exceeded" Nov 02 06:37:42.642057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.640153 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:42.663109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.655713 5025 generic.go:356] "Generic (PLEG): container finished" podID="849e78b5-270f-408d-a9be-931b9759e29f" containerID="b2c821b8ed0976c74b40d1bd016375863f13015530597e24e392881b41bb895b" exitCode=137 Nov 02 06:37:42.663109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.655948 5025 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f118d6027418a96671f73ecd825c7da5611cc89af58603fa22fab6577741bec" Nov 02 06:37:42.675823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.675670 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-948182" Nov 02 06:37:42.699153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.688894 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-721487" event={"ID":"8f22d5c9-c5b6-4e48-acf1-ab44f69925a5","Type":"ContainerStarted","Data":"e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe"} Nov 02 06:37:42.772312 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.772267 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849e78b5-270f-408d-a9be-931b9759e29f-kube-api-access-8vrp9\" (UniqueName: \"kubernetes.io/projected/849e78b5-270f-408d-a9be-931b9759e29f-kube-api-access-8vrp9\") pod \"849e78b5-270f-408d-a9be-931b9759e29f\" (UID: \"849e78b5-270f-408d-a9be-931b9759e29f\") " Nov 02 06:37:42.800668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.800631 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849e78b5-270f-408d-a9be-931b9759e29f-kube-api-access-8vrp9" pod "849e78b5-270f-408d-a9be-931b9759e29f" (UID: "849e78b5-270f-408d-a9be-931b9759e29f"). InnerVolumeSpecName "kube-api-access-8vrp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:42.812109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.803584 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7135-721487" podStartSLOduration=3.8035695499999997 podStartE2EDuration="3.80356955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:42.803395329 +0000 UTC m=+1239.251772086" watchObservedRunningTime="2026-11-02 06:37:42.80356955 +0000 UTC m=+1239.251946308" Nov 02 06:37:42.919177 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.919142 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vrp9\" (UniqueName: \"kubernetes.io/projected/849e78b5-270f-408d-a9be-931b9759e29f-kube-api-access-8vrp9\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:42.929234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:42.929192 5025 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.077162 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.076594 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.100103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.095660 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 06:37:43.100103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.095868 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:37:43.100103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.095930 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:37:43.167122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.148121 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.167122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.148507 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:37:43.174530 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.174475 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.177006 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.175245 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:37:43.186119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.181275 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.188297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.186382 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:37:43.325393 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.325358 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:43.325672 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.325633 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.397495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.397403 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:43.478605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.398943 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:43.594100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.565050 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:37:43.721127 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.653019 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 02 06:37:43.739109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.735798 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:43.739109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.735860 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.817119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.792221 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:43.851102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.847602 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:43.851626 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.846953 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-948182" Nov 02 06:37:43.902122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:43.878434 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:44.179163 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.132254 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded" Nov 02 06:37:44.179163 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.171268 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.206164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.180015 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 06:37:44.207820 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.204327 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.208247 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.204376 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.718272 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.718214 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.765100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.729148 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849e78b5-270f-408d-a9be-931b9759e29f" path="/var/lib/kubelet/pods/849e78b5-270f-408d-a9be-931b9759e29f/volumes" Nov 02 06:37:44.768703 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:37:44.768674 5025 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.091s" Nov 02 06:37:44.769115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.769065 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:37:44.795298 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.729555 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:44.796609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.796572 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:44.796891 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.796798 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:44.888157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.887177 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:44.888157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:44.887261 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.208102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:45.183127 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.208102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:45.183272 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded" Nov 02 06:37:45.253827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:45.210893 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.259228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:45.229976 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 06:37:45.259546 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:45.230018 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 06:37:45.561667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:45.561567 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:45.809916 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:45.809879 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:45.810248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:45.810216 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.810752 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:45.810220 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:45.980100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:45.943359 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:46.041100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:46.018313 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:46.440608 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:46.440574 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:46.582828 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:46.582797 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:37:46.735150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:46.730552 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:37:46.735150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:46.731920 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:46.741834 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:46.741813 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:37:46.920122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:46.917927 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:46.920122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:46.918240 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 06:37:46.920122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:46.918445 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 06:37:47.023941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:46.991696 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:47.023941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:46.991777 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:47.141106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.140597 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:47.141106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.140670 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:47.491101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.472780 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:37:47.491101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.472854 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:37:47.677220 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.674120 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:47.828269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.827665 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:47.848397 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.842230 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v"] Nov 02 06:37:47.873033 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.872992 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:47.873537 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.873020 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:47.873861 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.873843 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:37:47.874009 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.873996 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:47.874256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.874240 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v"] Nov 02 06:37:47.926409 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:47.926381 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-d690ed79-openshift-default-dockercfg-nmpm8\"" Nov 02 06:37:48.014999 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.014955 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-data\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.015290 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.015267 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jx6f\" (UniqueName: \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-kube-api-access-5jx6f\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.015450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.015418 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-token\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.015549 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.015531 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-credential-socket\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.015607 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.015591 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-certs\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.015716 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.015702 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-podinfo\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.015751 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.015738 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istiod-ca-cert\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.015794 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.015780 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-socket\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.015873 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.015817 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-envoy\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.062921 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.062888 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:48.079814 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.079729 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7135-721487"] Nov 02 06:37:48.080144 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.080003 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7135-721487" podUID="8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" containerName="test-curl-7135-721487" containerID="cri-o://e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe" gracePeriod=2 Nov 02 06:37:48.083572 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.083550 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7135-721487"] Nov 02 06:37:48.117161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.116764 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jx6f\" (UniqueName: \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-kube-api-access-5jx6f\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.117161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.116837 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-token\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.117161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.116899 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-credential-socket\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.117161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.116940 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-certs\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.117161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.117007 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-podinfo\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.117161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.117036 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istiod-ca-cert\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.117161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.117093 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-socket\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.117161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.117124 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-envoy\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.117710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.117236 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-data\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.117710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.117666 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-data\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.117977 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.117937 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-certs\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.118695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.118669 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-credential-socket\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.119284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.119262 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istiod-ca-cert\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.120320 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.120224 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-socket\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.124230 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.123999 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-envoy\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.126946 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.126868 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-podinfo\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.175878 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.175845 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-token\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.176722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.176692 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jx6f\" (UniqueName: \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-kube-api-access-5jx6f\") pod \"e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.182229 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.182196 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:37:48.182374 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.182254 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:37:48.212228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.212196 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:48.236674 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.236647 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:48.262552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.262524 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:48.267721 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:37:48.267696 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-721487\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:48.531525 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.531494 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:37:48.531979 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.531953 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:37:48.554703 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:37:48.554658 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-721487\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:48.653881 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.653854 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v"] Nov 02 06:37:48.654527 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:37:48.654504 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4a4872_6bcb_4c47_b62c_7ddab59ce9e5.slice/crio-488bd7df8a3ece3802e89809e7bf09f1d11e09c04a6815d21de2a89ab9ba69e0 WatchSource:0}: Error finding container 488bd7df8a3ece3802e89809e7bf09f1d11e09c04a6815d21de2a89ab9ba69e0: Status 404 returned error can't find the container with id 488bd7df8a3ece3802e89809e7bf09f1d11e09c04a6815d21de2a89ab9ba69e0 Nov 02 06:37:48.656852 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.656816 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:48.656948 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.656892 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:48.656948 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.656937 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:48.857055 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:37:48.856958 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-721487\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:48.958927 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:48.958891 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:49.155130 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:49.155026 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:49.338254 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:49.338213 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:49.538556 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:49.538510 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" event={"ID":"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5","Type":"ContainerStarted","Data":"a9de961f214c4c99e3aaae9430966f6731cced5066f5fe8b1ae53044dd216f8a"} Nov 02 06:37:49.538765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:49.538571 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" event={"ID":"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5","Type":"ContainerStarted","Data":"488bd7df8a3ece3802e89809e7bf09f1d11e09c04a6815d21de2a89ab9ba69e0"} Nov 02 06:37:49.650111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:49.650031 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" podStartSLOduration=2.650016373 podStartE2EDuration="2.650016373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:49.649316299 +0000 UTC m=+1246.097693056" watchObservedRunningTime="2026-11-02 06:37:49.650016373 +0000 UTC m=+1246.098393139" Nov 02 06:37:49.697452 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:37:49.697409 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-721487\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:50.212845 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.212813 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:50.218213 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.218190 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:50.316205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.316173 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:50.332260 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.332239 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:50.437104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.437060 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58"] Nov 02 06:37:50.437640 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.437623 5025 cpu_manager.go:414] "containerMap: removing container" podUID="8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" containerName="test-curl-7135-721487" Nov 02 06:37:50.437687 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.437641 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" containerName="test-curl-7135-721487" Nov 02 06:37:50.437753 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.437744 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" containerName="test-curl-7135-721487" Nov 02 06:37:50.440861 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.440846 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.495587 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.495559 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-14448910-openshift-default-dockercfg-v79h8\"" Nov 02 06:37:50.498198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.498172 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f22d5c9-c5b6-4e48-acf1-ab44f69925a5-kube-api-access-klwxc\" (UniqueName: \"kubernetes.io/projected/8f22d5c9-c5b6-4e48-acf1-ab44f69925a5-kube-api-access-klwxc\") pod \"8f22d5c9-c5b6-4e48-acf1-ab44f69925a5\" (UID: \"8f22d5c9-c5b6-4e48-acf1-ab44f69925a5\") " Nov 02 06:37:50.502113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.500791 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f22d5c9-c5b6-4e48-acf1-ab44f69925a5-kube-api-access-klwxc" pod "8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" (UID: "8f22d5c9-c5b6-4e48-acf1-ab44f69925a5"). InnerVolumeSpecName "kube-api-access-klwxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:37:50.502113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.501213 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58"] Nov 02 06:37:50.523329 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.523302 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:50.545607 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.545577 5025 generic.go:356] "Generic (PLEG): container finished" podID="8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" containerID="e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe" exitCode=137 Nov 02 06:37:50.545748 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.545626 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:50.545748 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.545670 5025 scope.go:122] "RemoveContainer" containerID="e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe" Nov 02 06:37:50.545998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.545982 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:50.547147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.547125 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:37:50.558638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.557836 5025 scope.go:122] "RemoveContainer" containerID="e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe" Nov 02 06:37:50.558638 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:37:50.558245 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe\": container with ID starting with e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe not found: ID does not exist" containerID="e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe" Nov 02 06:37:50.558638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.558279 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe"} err="failed to get container status \"e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe\": rpc error: code = NotFound desc = could not find container \"e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe\": container with ID starting with e7c39928e259611a05c29d8c4a7d0ebe1f0826dfa1a12e8e31fac3520c1835fe not found: ID does not exist" Nov 02 06:37:50.626536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.626041 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-socket\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.626536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.626134 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-credential-socket\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.626536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.626165 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-podinfo\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.626536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.626222 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-envoy\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.626536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.626267 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-token\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.650103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.648299 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdjl\" (UniqueName: \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-kube-api-access-dzdjl\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.650103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.648414 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-certs\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.650103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.648463 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istiod-ca-cert\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.650103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.648514 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-data\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.650103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.649163 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klwxc\" (UniqueName: \"kubernetes.io/projected/8f22d5c9-c5b6-4e48-acf1-ab44f69925a5-kube-api-access-klwxc\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:37:50.726148 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.721989 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:50.751096 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:37:50.746314 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-721487\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" pod="opendatahub/test-curl-7135-721487" Nov 02 06:37:50.784211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.784125 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-socket\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.784402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.784215 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-credential-socket\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.805304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.784865 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-podinfo\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.805304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.801604 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-envoy\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.805304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.801683 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-token\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.805304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.784813 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-socket\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.833855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.819538 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdjl\" (UniqueName: \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-kube-api-access-dzdjl\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.833855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.819653 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-certs\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.833855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.819724 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istiod-ca-cert\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.833855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.819775 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-data\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.833855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.820222 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-data\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.833855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.820986 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-certs\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.833855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.801458 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-credential-socket\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.843541 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.843496 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-envoy\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.853412 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.853380 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-podinfo\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.853884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.853846 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istiod-ca-cert\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.890140 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.889897 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdjl\" (UniqueName: \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-kube-api-access-dzdjl\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:50.906110 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.906011 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:50.915682 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:50.915629 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-token\") pod \"e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:51.054689 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:51.054607 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:51.245114 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:51.244594 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:51.522324 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:51.521708 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58"] Nov 02 06:37:51.524280 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:51.524053 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:51.524280 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:51.524185 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:51.524982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:51.524928 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:37:51.537398 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:51.537373 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:51.556217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:51.556188 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" event={"ID":"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc","Type":"ContainerStarted","Data":"f1039338865c449bada0df3d28e6e835f61f1694fd8926b87e50e7bbf50df934"} Nov 02 06:37:51.672196 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:51.672157 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f22d5c9-c5b6-4e48-acf1-ab44f69925a5" path="/var/lib/kubelet/pods/8f22d5c9-c5b6-4e48-acf1-ab44f69925a5/volumes" Nov 02 06:37:52.001098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:51.999623 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:52.562885 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:52.562846 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" event={"ID":"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc","Type":"ContainerStarted","Data":"870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af"} Nov 02 06:37:52.730918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:52.730865 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" podStartSLOduration=2.730852267 podStartE2EDuration="2.730852267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:52.729259343 +0000 UTC m=+1249.177636099" watchObservedRunningTime="2026-11-02 06:37:52.730852267 +0000 UTC m=+1249.179229022" Nov 02 06:37:52.960425 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:52.960395 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:53.057389 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.055551 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:53.068264 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.067526 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:53.264456 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.264376 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:53.499785 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.499757 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:53.698846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.698810 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:53.748098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.740036 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:53.748098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.740163 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:37:53.848267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.848230 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2"] Nov 02 06:37:53.852479 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.852454 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:37:53.901566 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.901536 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2"] Nov 02 06:37:53.906355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.904594 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5tsj\" (UniqueName: \"kubernetes.io/projected/c059158d-4a54-4b16-9431-e16b8d5e5772-kube-api-access-m5tsj\") pod \"payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2\" (UID: \"c059158d-4a54-4b16-9431-e16b8d5e5772\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:37:53.906355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.904886 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c059158d-4a54-4b16-9431-e16b8d5e5772-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2\" (UID: \"c059158d-4a54-4b16-9431-e16b8d5e5772\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:37:53.910573 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.910544 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-15867003\"" Nov 02 06:37:53.999480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:53.999389 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-15867003-dockercfg-zm9j6\"" Nov 02 06:37:54.078260 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.078221 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5tsj\" (UniqueName: \"kubernetes.io/projected/c059158d-4a54-4b16-9431-e16b8d5e5772-kube-api-access-m5tsj\") pod \"payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2\" (UID: \"c059158d-4a54-4b16-9431-e16b8d5e5772\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:37:54.078674 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.078648 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c059158d-4a54-4b16-9431-e16b8d5e5772-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2\" (UID: \"c059158d-4a54-4b16-9431-e16b8d5e5772\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:37:54.088050 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.088009 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c059158d-4a54-4b16-9431-e16b8d5e5772-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2\" (UID: \"c059158d-4a54-4b16-9431-e16b8d5e5772\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:37:54.095326 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.095132 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv"] Nov 02 06:37:54.121125 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.119518 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:37:54.130054 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.125925 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv"] Nov 02 06:37:54.169558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.166209 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5tsj\" (UniqueName: \"kubernetes.io/projected/c059158d-4a54-4b16-9431-e16b8d5e5772-kube-api-access-m5tsj\") pod \"payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2\" (UID: \"c059158d-4a54-4b16-9431-e16b8d5e5772\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:37:54.169558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.166602 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:37:54.209844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.208419 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca3fb1a-d179-4fd4-be33-d687a9d36926-plugins-config-volume\") pod \"payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv\" (UID: \"5ca3fb1a-d179-4fd4-be33-d687a9d36926\") " pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:37:54.209844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.208644 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptjmn\" (UniqueName: \"kubernetes.io/projected/5ca3fb1a-d179-4fd4-be33-d687a9d36926-kube-api-access-ptjmn\") pod \"payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv\" (UID: \"5ca3fb1a-d179-4fd4-be33-d687a9d36926\") " pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:37:54.310723 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.310599 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptjmn\" (UniqueName: \"kubernetes.io/projected/5ca3fb1a-d179-4fd4-be33-d687a9d36926-kube-api-access-ptjmn\") pod \"payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv\" (UID: \"5ca3fb1a-d179-4fd4-be33-d687a9d36926\") " pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:37:54.310882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.310732 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca3fb1a-d179-4fd4-be33-d687a9d36926-plugins-config-volume\") pod \"payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv\" (UID: \"5ca3fb1a-d179-4fd4-be33-d687a9d36926\") " pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:37:54.311671 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.311644 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca3fb1a-d179-4fd4-be33-d687a9d36926-plugins-config-volume\") pod \"payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv\" (UID: \"5ca3fb1a-d179-4fd4-be33-d687a9d36926\") " pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:37:54.367166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.367132 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptjmn\" (UniqueName: \"kubernetes.io/projected/5ca3fb1a-d179-4fd4-be33-d687a9d36926-kube-api-access-ptjmn\") pod \"payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv\" (UID: \"5ca3fb1a-d179-4fd4-be33-d687a9d36926\") " pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:37:54.507482 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.507454 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:37:54.608971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.608934 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2"] Nov 02 06:37:54.615989 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:37:54.615961 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc059158d_4a54_4b16_9431_e16b8d5e5772.slice/crio-d5e0f9ba1160972706e93b106677a8d13522ec7bea4caa1b7d70638294ff6271 WatchSource:0}: Error finding container d5e0f9ba1160972706e93b106677a8d13522ec7bea4caa1b7d70638294ff6271: Status 404 returned error can't find the container with id d5e0f9ba1160972706e93b106677a8d13522ec7bea4caa1b7d70638294ff6271 Nov 02 06:37:54.730787 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.730754 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" event={"ID":"c059158d-4a54-4b16-9431-e16b8d5e5772","Type":"ContainerStarted","Data":"b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8"} Nov 02 06:37:54.731131 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.730811 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" event={"ID":"c059158d-4a54-4b16-9431-e16b8d5e5772","Type":"ContainerStarted","Data":"d5e0f9ba1160972706e93b106677a8d13522ec7bea4caa1b7d70638294ff6271"} Nov 02 06:37:54.731131 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.730952 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:37:54.840850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.840759 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" podStartSLOduration=1.840746283 podStartE2EDuration="1.840746283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:54.838806498 +0000 UTC m=+1251.287183266" watchObservedRunningTime="2026-11-02 06:37:54.840746283 +0000 UTC m=+1251.289123080" Nov 02 06:37:54.875453 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:54.875431 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv"] Nov 02 06:37:54.876053 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:37:54.876028 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ca3fb1a_d179_4fd4_be33_d687a9d36926.slice/crio-d23cfabd0ea918831417a472a72aab6710d097a50dd1665881a5fa2d7220b6c6 WatchSource:0}: Error finding container d23cfabd0ea918831417a472a72aab6710d097a50dd1665881a5fa2d7220b6c6: Status 404 returned error can't find the container with id d23cfabd0ea918831417a472a72aab6710d097a50dd1665881a5fa2d7220b6c6 Nov 02 06:37:55.744910 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:55.744858 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" event={"ID":"5ca3fb1a-d179-4fd4-be33-d687a9d36926","Type":"ContainerStarted","Data":"0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee"} Nov 02 06:37:55.745406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:55.744954 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" event={"ID":"5ca3fb1a-d179-4fd4-be33-d687a9d36926","Type":"ContainerStarted","Data":"d23cfabd0ea918831417a472a72aab6710d097a50dd1665881a5fa2d7220b6c6"} Nov 02 06:37:55.745406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:55.745035 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:37:55.856586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:55.856527 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" podStartSLOduration=2.856512749 podStartE2EDuration="2.856512749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:55.85428947 +0000 UTC m=+1252.302666227" watchObservedRunningTime="2026-11-02 06:37:55.856512749 +0000 UTC m=+1252.304889515" Nov 02 06:37:56.715256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:56.715227 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:57.015278 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:57.015198 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:57.225129 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:57.220735 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:58.088665 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.088589 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:58.288681 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.288645 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:58.451214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.451179 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx"] Nov 02 06:37:58.460134 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.460095 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:37:58.475897 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.475870 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:37:58.505270 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.505244 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx"] Nov 02 06:37:58.562518 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.562480 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlz6v\" (UniqueName: \"kubernetes.io/projected/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-kube-api-access-qlz6v\") pod \"payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx\" (UID: \"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:37:58.567096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.562800 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx\" (UID: \"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:37:58.613097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.612390 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d690ed79\"" Nov 02 06:37:58.674508 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.674475 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z"] Nov 02 06:37:58.698651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.698535 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlz6v\" (UniqueName: \"kubernetes.io/projected/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-kube-api-access-qlz6v\") pod \"payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx\" (UID: \"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:37:58.699004 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.698976 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx\" (UID: \"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:37:58.731304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.702544 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx\" (UID: \"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:37:58.734216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.734188 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z"] Nov 02 06:37:58.734358 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.734309 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:37:58.743466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.742492 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d690ed79-dockercfg-fwt7w\"" Nov 02 06:37:58.758316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.757215 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlz6v\" (UniqueName: \"kubernetes.io/projected/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-kube-api-access-qlz6v\") pod \"payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx\" (UID: \"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:37:58.777455 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.777425 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:37:58.833945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.833359 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wftpb\" (UniqueName: \"kubernetes.io/projected/7bf38d06-2b67-4c71-9924-1033ebdad32e-kube-api-access-wftpb\") pod \"payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z\" (UID: \"7bf38d06-2b67-4c71-9924-1033ebdad32e\") " pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:37:58.833945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.833621 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf38d06-2b67-4c71-9924-1033ebdad32e-plugins-config-volume\") pod \"payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z\" (UID: \"7bf38d06-2b67-4c71-9924-1033ebdad32e\") " pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:37:58.934988 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.934954 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wftpb\" (UniqueName: \"kubernetes.io/projected/7bf38d06-2b67-4c71-9924-1033ebdad32e-kube-api-access-wftpb\") pod \"payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z\" (UID: \"7bf38d06-2b67-4c71-9924-1033ebdad32e\") " pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:37:58.935183 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.935148 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf38d06-2b67-4c71-9924-1033ebdad32e-plugins-config-volume\") pod \"payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z\" (UID: \"7bf38d06-2b67-4c71-9924-1033ebdad32e\") " pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:37:58.935797 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.935778 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf38d06-2b67-4c71-9924-1033ebdad32e-plugins-config-volume\") pod \"payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z\" (UID: \"7bf38d06-2b67-4c71-9924-1033ebdad32e\") " pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:37:58.990045 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:58.989978 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wftpb\" (UniqueName: \"kubernetes.io/projected/7bf38d06-2b67-4c71-9924-1033ebdad32e-kube-api-access-wftpb\") pod \"payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z\" (UID: \"7bf38d06-2b67-4c71-9924-1033ebdad32e\") " pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:37:59.075427 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:59.075391 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:37:59.145605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:59.145575 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx"] Nov 02 06:37:59.146014 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:37:59.145950 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fcd0fc_f9ae_4ce1_bdfd_4e28dd4fad28.slice/crio-2e0d938fad2e54268e2987978cda48bf4547a8563468f37caa5295eadee1c54a WatchSource:0}: Error finding container 2e0d938fad2e54268e2987978cda48bf4547a8563468f37caa5295eadee1c54a: Status 404 returned error can't find the container with id 2e0d938fad2e54268e2987978cda48bf4547a8563468f37caa5295eadee1c54a Nov 02 06:37:59.444493 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:59.444468 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z"] Nov 02 06:37:59.445212 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:37:59.445186 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf38d06_2b67_4c71_9924_1033ebdad32e.slice/crio-7832d2123a3a3d0c680487ce4bc796ac2c9e2489fa720fdf03a34ea27cb5c84e WatchSource:0}: Error finding container 7832d2123a3a3d0c680487ce4bc796ac2c9e2489fa720fdf03a34ea27cb5c84e: Status 404 returned error can't find the container with id 7832d2123a3a3d0c680487ce4bc796ac2c9e2489fa720fdf03a34ea27cb5c84e Nov 02 06:37:59.771579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:59.771493 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" event={"ID":"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28","Type":"ContainerStarted","Data":"3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3"} Nov 02 06:37:59.771579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:59.771560 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:37:59.771778 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:59.771588 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" event={"ID":"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28","Type":"ContainerStarted","Data":"2e0d938fad2e54268e2987978cda48bf4547a8563468f37caa5295eadee1c54a"} Nov 02 06:37:59.773242 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:59.773211 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" event={"ID":"7bf38d06-2b67-4c71-9924-1033ebdad32e","Type":"ContainerStarted","Data":"934094c57bafdee88bd967233e73b0364b741b7adbc4eb1894fb338a7176f7d4"} Nov 02 06:37:59.773368 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:59.773260 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" event={"ID":"7bf38d06-2b67-4c71-9924-1033ebdad32e","Type":"ContainerStarted","Data":"7832d2123a3a3d0c680487ce4bc796ac2c9e2489fa720fdf03a34ea27cb5c84e"} Nov 02 06:37:59.773368 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:59.773279 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:37:59.881491 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:59.881439 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" podStartSLOduration=1.8814261939999999 podStartE2EDuration="1.881426194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:59.88110642 +0000 UTC m=+1256.329483192" watchObservedRunningTime="2026-11-02 06:37:59.881426194 +0000 UTC m=+1256.329802963" Nov 02 06:38:00.005103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:37:59.995464 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" podStartSLOduration=1.995444514 podStartE2EDuration="1.995444514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:37:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:37:59.989956092 +0000 UTC m=+1256.438332885" watchObservedRunningTime="2026-11-02 06:37:59.995444514 +0000 UTC m=+1256.443821270" Nov 02 06:38:00.479102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:00.479053 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:00.676580 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:00.676540 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:01.454544 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.454506 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj"] Nov 02 06:38:01.458913 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.458890 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.574641 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.574601 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p"] Nov 02 06:38:01.579189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.579162 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87ztf\" (UniqueName: \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-kube-api-access-87ztf\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.579319 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.579203 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e49f6b2-4f57-488f-a668-638741a40d60-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.579319 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.579243 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.579319 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.579258 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.579412 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.579384 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.579473 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.579425 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e49f6b2-4f57-488f-a668-638741a40d60-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.579554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.579535 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.579629 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.579577 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.579699 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.579639 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.579909 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.579883 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.582759 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.582735 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj"] Nov 02 06:38:01.629946 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.629911 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p"] Nov 02 06:38:01.681577 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.681547 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.681737 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.681587 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.681737 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.681623 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.681810 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.681749 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.681884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.681868 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.681917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.681899 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.681982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.681966 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6mf6\" (UniqueName: \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-kube-api-access-n6mf6\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.682034 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682003 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87ztf\" (UniqueName: \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-kube-api-access-87ztf\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.682102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682060 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.682158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682139 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-data\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.682253 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682140 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-credential-socket\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.682253 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682179 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e49f6b2-4f57-488f-a668-638741a40d60-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.682253 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682211 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.682253 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682251 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.682464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682279 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.682464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682359 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d047f344-8af2-4098-847c-a8e1447e7bc8-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.682464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682437 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.682620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682472 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e49f6b2-4f57-488f-a668-638741a40d60-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.682620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682500 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.682620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682524 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.682773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682638 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.682773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682739 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-certs\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.682946 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.682925 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e49f6b2-4f57-488f-a668-638741a40d60-istiod-ca-cert\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.684137 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.684122 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-envoy\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.684684 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.684658 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e49f6b2-4f57-488f-a668-638741a40d60-istio-podinfo\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.743938 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.743857 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-istio-token\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.744703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.744680 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87ztf\" (UniqueName: \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-kube-api-access-87ztf\") pod \"e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.773460 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.773432 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:01.783805 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.783765 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.783915 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.783889 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.783954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.783939 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6mf6\" (UniqueName: \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-kube-api-access-n6mf6\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.784005 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.783973 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.784063 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.784009 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.784063 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.784044 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d047f344-8af2-4098-847c-a8e1447e7bc8-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.784179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.784144 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.784179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.784172 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.784281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.784266 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.784340 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.784295 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.786139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.784500 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.786139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.784665 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.786139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.784721 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.786139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.784866 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d047f344-8af2-4098-847c-a8e1447e7bc8-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.786628 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.786606 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.786730 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.786712 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.856153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.847547 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.856153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.850928 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6mf6\" (UniqueName: \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-kube-api-access-n6mf6\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.856153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.851145 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf"] Nov 02 06:38:01.857434 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.857413 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:01.907530 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.907166 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:01.918134 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.918103 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf"] Nov 02 06:38:01.986768 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.986739 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:01.986899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.986777 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:01.986899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.986803 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjth7\" (UniqueName: \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-kube-api-access-zjth7\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:01.987017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.986937 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:01.987017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.986958 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:01.987017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.986976 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:01.987017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.986997 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab885c6e-6f00-421e-8ca1-313e33e7efba-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:01.987017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.987014 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:01.987346 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:01.987056 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.088933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.088846 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.088933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.088892 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.088933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.088919 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.089289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.088948 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab885c6e-6f00-421e-8ca1-313e33e7efba-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.089289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.088973 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.089289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.089028 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.089289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.089157 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.089289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.089187 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.089289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.089222 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjth7\" (UniqueName: \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-kube-api-access-zjth7\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.089558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.089352 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.089558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.089409 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.089667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.089647 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.089704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.089687 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.089841 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.089819 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab885c6e-6f00-421e-8ca1-313e33e7efba-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.091330 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.091310 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.091555 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.091527 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.143869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.143832 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.143979 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.143964 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjth7\" (UniqueName: \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-kube-api-access-zjth7\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.173428 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.173402 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:02.195443 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.195419 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj"] Nov 02 06:38:02.198046 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:38:02.198018 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e49f6b2_4f57_488f_a668_638741a40d60.slice/crio-81ae8acb0c34e777e922fc790188783787257f27e93f1865faa94f8add632c6c WatchSource:0}: Error finding container 81ae8acb0c34e777e922fc790188783787257f27e93f1865faa94f8add632c6c: Status 404 returned error can't find the container with id 81ae8acb0c34e777e922fc790188783787257f27e93f1865faa94f8add632c6c Nov 02 06:38:02.200721 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.200689 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.200813 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.200771 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.200872 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.200815 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.267353 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.267272 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:02.304048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.304006 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p"] Nov 02 06:38:02.331280 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.331095 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.337126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.331508 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.337126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.331573 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.565173 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.561761 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf"] Nov 02 06:38:02.565173 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.562056 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.565173 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.562380 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.565173 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.562484 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:02.606478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.606378 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:02.832750 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.832221 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:02.952918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.952874 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" event={"ID":"d047f344-8af2-4098-847c-a8e1447e7bc8","Type":"ContainerStarted","Data":"9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce"} Nov 02 06:38:02.953239 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:02.953211 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" event={"ID":"d047f344-8af2-4098-847c-a8e1447e7bc8","Type":"ContainerStarted","Data":"b1105775f01b027e9b7800e2557ab7af2b19e350471362876e44fa443e8bffd9"} Nov 02 06:38:03.070100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.067501 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4"] Nov 02 06:38:03.126201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.090353 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:38:03.126201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.091413 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4"] Nov 02 06:38:03.126201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.097419 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" event={"ID":"4e49f6b2-4f57-488f-a668-638741a40d60","Type":"ContainerStarted","Data":"6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3"} Nov 02 06:38:03.126201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.097490 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" event={"ID":"4e49f6b2-4f57-488f-a668-638741a40d60","Type":"ContainerStarted","Data":"81ae8acb0c34e777e922fc790188783787257f27e93f1865faa94f8add632c6c"} Nov 02 06:38:03.126201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.115111 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" event={"ID":"ab885c6e-6f00-421e-8ca1-313e33e7efba","Type":"ContainerStarted","Data":"5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911"} Nov 02 06:38:03.126201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.115192 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" event={"ID":"ab885c6e-6f00-421e-8ca1-313e33e7efba","Type":"ContainerStarted","Data":"6ed74f23456112f15fa3db1660630dd7bfc5450f436f04c882cb7b396d8ebaff"} Nov 02 06:38:03.144575 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.144535 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5eed1bfb\"" Nov 02 06:38:03.149748 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.149717 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5eed1bfb-dockercfg-rpwf4\"" Nov 02 06:38:03.332141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.209140 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:03.332141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.238129 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4\" (UID: \"13e5782e-4b0f-4c0c-8707-c1efa3f6c756\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:38:03.332141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.324346 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:38:03.332141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.332067 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 06:38:03.353742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.332147 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hrnj\" (UniqueName: \"kubernetes.io/projected/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-kube-api-access-4hrnj\") pod \"payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4\" (UID: \"13e5782e-4b0f-4c0c-8707-c1efa3f6c756\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:38:03.353742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.334365 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc"] Nov 02 06:38:03.353742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.337571 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" podStartSLOduration=2.337553834 podStartE2EDuration="2.337553834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:03.094813032 +0000 UTC m=+1259.543189788" watchObservedRunningTime="2026-11-02 06:38:03.337553834 +0000 UTC m=+1259.785930617" Nov 02 06:38:03.359158 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:38:03.358993 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:38:03.374156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.373057 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc"] Nov 02 06:38:03.374156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.373213 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.432119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.431774 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podStartSLOduration=2.431758859 podStartE2EDuration="2.431758859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:03.394858962 +0000 UTC m=+1259.843235734" watchObservedRunningTime="2026-11-02 06:38:03.431758859 +0000 UTC m=+1259.880135616" Nov 02 06:38:03.433845 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.433654 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4\" (UID: \"13e5782e-4b0f-4c0c-8707-c1efa3f6c756\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:38:03.433845 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.433721 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hrnj\" (UniqueName: \"kubernetes.io/projected/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-kube-api-access-4hrnj\") pod \"payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4\" (UID: \"13e5782e-4b0f-4c0c-8707-c1efa3f6c756\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:38:03.435938 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.435911 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4\" (UID: \"13e5782e-4b0f-4c0c-8707-c1efa3f6c756\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:38:03.524103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.520826 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" podStartSLOduration=2.520806048 podStartE2EDuration="2.520806048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:03.519577326 +0000 UTC m=+1259.967954110" watchObservedRunningTime="2026-11-02 06:38:03.520806048 +0000 UTC m=+1259.969182804" Nov 02 06:38:03.533130 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.531876 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hrnj\" (UniqueName: \"kubernetes.io/projected/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-kube-api-access-4hrnj\") pod \"payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4\" (UID: \"13e5782e-4b0f-4c0c-8707-c1efa3f6c756\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:38:03.557108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.535774 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.557108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.535836 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.557108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.535925 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.557108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.535965 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.557108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.535997 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.557108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.536026 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9sl6\" (UniqueName: \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-kube-api-access-g9sl6\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.557108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.536173 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.557108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.536273 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.557108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.536367 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.653342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.653255 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.653900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.653871 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.654353 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.653913 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.654353 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.653943 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9sl6\" (UniqueName: \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-kube-api-access-g9sl6\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.654353 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.654050 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.654901 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.654160 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.656884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.654368 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.656884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.654569 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.656884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.654807 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.656884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.656632 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.657111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.656814 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.657222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.657212 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.657332 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.657322 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.657586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.657235 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.657704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.657623 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.659634 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.659611 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.749100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.748763 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.752017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.751976 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9sl6\" (UniqueName: \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-kube-api-access-g9sl6\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:03.756462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.756420 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5eed1bfb-dockercfg-rpwf4\"" Nov 02 06:38:03.758680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.758657 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:38:03.773661 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.773635 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:03.774925 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.774898 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 06:38:03.775020 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.774948 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 06:38:03.956097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.929141 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:03.956097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.934184 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 06:38:03.956097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:03.934240 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 06:38:04.025099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.017491 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:04.174701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.174667 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:38:04.174862 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.174716 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 06:38:04.361099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.361055 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4"] Nov 02 06:38:04.574812 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.574778 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc"] Nov 02 06:38:04.578058 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:38:04.577966 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e926c7_f29a_4ad2_b404_b1920c3a15d7.slice/crio-a1465ba52c391fd51babeb1e1cf5d58b42bf23ac5c8b2cd7a4c351ff54418741 WatchSource:0}: Error finding container a1465ba52c391fd51babeb1e1cf5d58b42bf23ac5c8b2cd7a4c351ff54418741: Status 404 returned error can't find the container with id a1465ba52c391fd51babeb1e1cf5d58b42bf23ac5c8b2cd7a4c351ff54418741 Nov 02 06:38:04.580711 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.580675 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:04.580804 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.580741 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:04.580804 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.580774 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:04.773902 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.773868 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 06:38:04.774283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.773915 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 06:38:04.930115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.920471 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 06:38:04.930115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:04.920536 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 06:38:05.174389 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.173917 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:38:05.174389 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.173978 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 06:38:05.179714 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.179679 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" event={"ID":"18e926c7-f29a-4ad2-b404-b1920c3a15d7","Type":"ContainerStarted","Data":"3ceca20941b1cf883e559f096c46a85f6104d2fdad8a5d30c7146296636abbc5"} Nov 02 06:38:05.179846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.179738 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" event={"ID":"18e926c7-f29a-4ad2-b404-b1920c3a15d7","Type":"ContainerStarted","Data":"a1465ba52c391fd51babeb1e1cf5d58b42bf23ac5c8b2cd7a4c351ff54418741"} Nov 02 06:38:05.183026 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.183005 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" event={"ID":"13e5782e-4b0f-4c0c-8707-c1efa3f6c756","Type":"ContainerStarted","Data":"8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb"} Nov 02 06:38:05.183114 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.183045 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:38:05.183114 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.183060 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" event={"ID":"13e5782e-4b0f-4c0c-8707-c1efa3f6c756","Type":"ContainerStarted","Data":"b697521bd62c217f129f5f13fc055dec3d21e206322a0d88b3dce1a13347efc5"} Nov 02 06:38:05.347102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.339241 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" podStartSLOduration=2.339221269 podStartE2EDuration="2.339221269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:05.31167495 +0000 UTC m=+1261.760051708" watchObservedRunningTime="2026-11-02 06:38:05.339221269 +0000 UTC m=+1261.787598025" Nov 02 06:38:05.639161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.638320 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:05.784536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.784501 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 06:38:05.785127 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.785092 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 06:38:05.966212 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.966178 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.023106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:05.992571 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 06:38:06.066132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.041262 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:06.066132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.041991 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" podStartSLOduration=4.041973568 podStartE2EDuration="4.041973568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:05.430372714 +0000 UTC m=+1261.878749498" watchObservedRunningTime="2026-11-02 06:38:06.041973568 +0000 UTC m=+1262.490350327" Nov 02 06:38:06.076464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.075640 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.076464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.075705 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 06:38:06.090640 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.090608 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:06.090904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.090890 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds"] Nov 02 06:38:06.340152 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.334958 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded" Nov 02 06:38:06.388255 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.350054 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.388255 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.350135 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 06:38:06.414111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.405171 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:06.414111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.413406 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:38:06.414395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.414365 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:38:06.465269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.445331 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds"] Nov 02 06:38:06.503101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.490481 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-14448910-dockercfg-59cr5\"" Nov 02 06:38:06.503101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.490580 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-14448910\"" Nov 02 06:38:06.679561 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.667175 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1516cf9d-8103-4605-af30-fae2e93df166-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds\" (UID: \"1516cf9d-8103-4605-af30-fae2e93df166\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:38:06.679561 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.667312 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp7nj\" (UniqueName: \"kubernetes.io/projected/1516cf9d-8103-4605-af30-fae2e93df166-kube-api-access-gp7nj\") pod \"payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds\" (UID: \"1516cf9d-8103-4605-af30-fae2e93df166\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:38:06.822111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.770257 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1516cf9d-8103-4605-af30-fae2e93df166-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds\" (UID: \"1516cf9d-8103-4605-af30-fae2e93df166\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:38:06.822111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.770377 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp7nj\" (UniqueName: \"kubernetes.io/projected/1516cf9d-8103-4605-af30-fae2e93df166-kube-api-access-gp7nj\") pod \"payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds\" (UID: \"1516cf9d-8103-4605-af30-fae2e93df166\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:38:06.822111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.781648 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 06:38:06.822111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.781702 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 06:38:06.856343 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.823698 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1516cf9d-8103-4605-af30-fae2e93df166-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds\" (UID: \"1516cf9d-8103-4605-af30-fae2e93df166\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:38:07.116935 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:06.963958 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:07.116935 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.007568 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:07.116935 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.045307 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp7nj\" (UniqueName: \"kubernetes.io/projected/1516cf9d-8103-4605-af30-fae2e93df166-kube-api-access-gp7nj\") pod \"payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds\" (UID: \"1516cf9d-8103-4605-af30-fae2e93df166\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:38:07.116935 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.047011 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 06:38:07.116935 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.056442 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 06:38:07.116935 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.056498 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 06:38:07.116935 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.047063 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 06:38:07.116935 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.082372 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:38:07.218100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.215644 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:38:07.218100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.215704 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 06:38:07.218100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.216994 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:07.494283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.400169 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:07.494283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.408377 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:07.494283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.409122 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:38:07.494283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.424159 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:38:07.908135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.850219 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 06:38:07.908135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:07.850283 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 06:38:08.167167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.167026 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 06:38:08.167595 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.167432 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 06:38:08.170060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.169944 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 06:38:08.170201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.170094 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 06:38:08.231102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.193765 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:38:08.231102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.193844 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 06:38:08.231102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.194609 5025 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.280096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.269357 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.498124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.466345 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.583120 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.513280 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.583120 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.540994 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:38:08.814150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.759056 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:08.814150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.808961 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:08.814150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.809316 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 06:38:08.814150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:08.809371 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 06:38:09.016188 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.004023 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.065266 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.029147 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 06:38:09.065266 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.023469 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.065266 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.029239 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 06:38:09.100775 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.099195 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.236145 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.232608 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.236422 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.236376 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 06:38:09.289358 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.280068 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.289358 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.280418 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:38:09.291020 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.290984 5025 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.309638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.261129 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 06:38:09.596100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.595917 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 06:38:09.660100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.650341 5025 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": context deadline exceeded" Nov 02 06:38:09.804097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.787221 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.869161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.813609 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:09.869161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.815742 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 06:38:09.869161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:09.815792 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 06:38:10.094116 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:38:10.034255 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1516cf9d_8103_4605_af30_fae2e93df166.slice/crio-2e8e5e640bcb7662b5705e92b8bbf3da1e6db1290b11940c0fd5f20e36263550 WatchSource:0}: Error finding container 2e8e5e640bcb7662b5705e92b8bbf3da1e6db1290b11940c0fd5f20e36263550: Status 404 returned error can't find the container with id 2e8e5e640bcb7662b5705e92b8bbf3da1e6db1290b11940c0fd5f20e36263550 Nov 02 06:38:10.094116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.034751 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 06:38:10.094116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.034796 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 06:38:10.098709 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.098671 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds"] Nov 02 06:38:10.098993 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.098975 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:38:10.109094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.099146 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:38:10.156294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.156197 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:10.157322 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.157297 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:38:10.264104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.263147 5025 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:10.362216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.362169 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:10.394388 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.362597 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:38:10.394388 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.393420 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 06:38:10.394388 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.393533 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:38:10.457662 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.457624 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 06:38:10.606598 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.606554 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 06:38:10.645098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.636832 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:10.792518 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.792425 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:10.795029 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.795000 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 06:38:10.817123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.817066 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 06:38:10.817344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.800053 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" event={"ID":"1516cf9d-8103-4605-af30-fae2e93df166","Type":"ContainerStarted","Data":"2e8e5e640bcb7662b5705e92b8bbf3da1e6db1290b11940c0fd5f20e36263550"} Nov 02 06:38:10.875488 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.873192 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:10.889122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.883388 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:38:10.946144 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.942402 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 06:38:10.946427 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.946395 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 06:38:10.959272 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.959240 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:38:10.983431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.983404 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:38:10.985720 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:10.985703 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:38:11.038101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.034894 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:38:11.058346 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.045681 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.058583 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.058549 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 06:38:11.078040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.074945 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:38:11.078040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.077984 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:38:11.177448 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.177412 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.178064 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.178034 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 06:38:11.774406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.774369 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.774695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.774663 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" Nov 02 06:38:11.843543 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.843499 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" event={"ID":"1516cf9d-8103-4605-af30-fae2e93df166","Type":"ContainerStarted","Data":"697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1"} Nov 02 06:38:11.843737 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.843571 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:38:11.938916 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.938878 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Nov 02 06:38:11.939139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:11.938935 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Nov 02 06:38:12.022107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:12.018812 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 06:38:12.022107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:12.018874 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 06:38:12.174567 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:12.174524 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body= Nov 02 06:38:12.174758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:12.174590 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" Nov 02 06:38:12.261133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:12.252884 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" podStartSLOduration=7.252859634 podStartE2EDuration="7.252859634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:12.23927496 +0000 UTC m=+1268.687651741" watchObservedRunningTime="2026-11-02 06:38:12.252859634 +0000 UTC m=+1268.701236391" Nov 02 06:38:13.039268 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:13.039229 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 06:38:13.039557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:13.039526 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 06:38:13.113101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:13.104045 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:13.113101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:13.105573 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:13.130100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:13.118271 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:38:13.327914 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:13.327818 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:13.328478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:13.328457 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:13.329308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:13.329285 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:38:13.393645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:13.392846 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:13.393645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:13.393372 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:13.402578 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:13.402552 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:38:14.019513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:14.019424 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Nov 02 06:38:14.019513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:14.019493 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Nov 02 06:38:14.188838 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:14.188781 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:15.022374 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:15.022344 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:15.022819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:15.022627 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:15.023331 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:15.023313 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:38:15.351299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:15.351227 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:16.446560 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.446533 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:38:16.684573 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.684535 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm"] Nov 02 06:38:16.689048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.689028 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.741973 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.741893 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm"] Nov 02 06:38:16.759915 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.759884 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.760109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.759985 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.760109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.760101 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.760220 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.760202 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.760266 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.760255 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.760439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.760414 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c758e46-4a47-4aa9-abee-798b5c3c0111-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.760488 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.760464 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.760586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.760569 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.760625 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.760600 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-944zh\" (UniqueName: \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-kube-api-access-944zh\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.862494 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.862460 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.862686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.862502 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-944zh\" (UniqueName: \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-kube-api-access-944zh\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.862686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.862595 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.862686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.862643 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.862848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.862700 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.862848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.862766 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.862848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.862809 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.863000 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.862915 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c758e46-4a47-4aa9-abee-798b5c3c0111-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.863000 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.862947 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.863135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.863006 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-data\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.863135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.863042 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-credential-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.863263 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.863239 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-socket\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.863311 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.863242 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-certs\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.863594 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.863574 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c758e46-4a47-4aa9-abee-798b5c3c0111-istiod-ca-cert\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.865014 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.864992 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-envoy\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.865366 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.865349 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-podinfo\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.919149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.919123 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-token\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:16.919259 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:16.919151 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-944zh\" (UniqueName: \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-kube-api-access-944zh\") pod \"e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:17.010228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:17.010119 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:17.448661 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:17.448628 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm"] Nov 02 06:38:17.482532 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:38:17.482502 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c758e46_4a47_4aa9_abee_798b5c3c0111.slice/crio-01e1e056a56b4f438864d204455fe448cea866ff4024e69f285b78b5818a3396 WatchSource:0}: Error finding container 01e1e056a56b4f438864d204455fe448cea866ff4024e69f285b78b5818a3396: Status 404 returned error can't find the container with id 01e1e056a56b4f438864d204455fe448cea866ff4024e69f285b78b5818a3396 Nov 02 06:38:17.485280 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:17.485239 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:17.485373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:17.485330 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:17.485373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:17.485364 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:17.906935 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:17.906852 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" event={"ID":"4c758e46-4a47-4aa9-abee-798b5c3c0111","Type":"ContainerStarted","Data":"8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9"} Nov 02 06:38:17.906935 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:17.906924 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" event={"ID":"4c758e46-4a47-4aa9-abee-798b5c3c0111","Type":"ContainerStarted","Data":"01e1e056a56b4f438864d204455fe448cea866ff4024e69f285b78b5818a3396"} Nov 02 06:38:18.015099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:18.014595 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:18.019099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:18.016760 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 06:38:18.019099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:18.016818 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 06:38:18.024361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:18.024317 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podStartSLOduration=2.024303137 podStartE2EDuration="2.024303137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:18.021882239 +0000 UTC m=+1274.470258994" watchObservedRunningTime="2026-11-02 06:38:18.024303137 +0000 UTC m=+1274.472679893" Nov 02 06:38:18.815154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:18.815116 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:19.014501 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:19.014469 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 06:38:19.014677 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:19.014538 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 06:38:20.023096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:20.022385 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 06:38:20.023096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:20.022448 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 06:38:20.652102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:20.647186 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:20.707101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:20.704656 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:20.935100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:20.930239 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:20.993303 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:20.993235 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.023543 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:21.023507 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 06:38:21.024130 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:21.024096 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 06:38:21.086216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:21.086167 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.120193 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:21.120146 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.655099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:21.647355 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:21.655099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:21.648179 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:38:21.936110 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:21.916802 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:21.964256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:21.937575 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.006115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:21.997189 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:38:22.272198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:22.043923 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.272198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:22.044333 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:38:22.272198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:22.045892 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 06:38:22.272198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:22.045931 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 06:38:22.272198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:22.166676 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.308683 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:22.308637 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Nov 02 06:38:22.310214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:22.310191 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:38:22.657155 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:22.656491 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:22.972227 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:22.955359 5025 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.015103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:22.976140 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.015103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.008479 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:38:23.015103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.011371 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:38:23.066593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.066544 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": context deadline exceeded" Nov 02 06:38:23.071320 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.069255 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 06:38:23.071320 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.069308 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 06:38:23.191102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.188250 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.272296 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.246285 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:38:23.330348 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.330256 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:23.437108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.434956 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" Nov 02 06:38:23.484114 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.484069 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:38:23.667014 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.666975 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": context deadline exceeded" Nov 02 06:38:23.805299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.803967 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:38:23.805595 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:23.805576 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 06:38:24.037170 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:24.037070 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 06:38:24.037408 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:24.037381 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 06:38:25.023847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:25.023812 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:25.028222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:25.027550 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 06:38:25.028446 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:25.028419 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 06:38:25.229742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:25.229370 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:26.014600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:26.014520 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 06:38:26.014803 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:26.014636 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 06:38:26.875068 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:26.875005 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:27.014019 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:27.013987 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body= Nov 02 06:38:27.014199 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:27.014038 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" Nov 02 06:38:27.066352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:27.066316 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:27.265568 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:27.265535 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:29.056222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:29.056185 5025 patch_prober.go:29] interesting pod/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:38:29.056618 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:29.056254 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:38:29.544260 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:29.544207 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:29.544974 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:29.544949 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:29.545665 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:29.545647 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:38:29.679613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:29.679573 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:31.048549 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:31.048518 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:32.707399 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:38:32.707361 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:38:33.272590 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.272539 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9"] Nov 02 06:38:33.278139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.278117 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.316871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.316838 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.317008 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.316945 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.317008 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.316973 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.317119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.317098 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbgs2\" (UniqueName: \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-kube-api-access-cbgs2\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.317171 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.317134 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.317171 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.317154 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.317282 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.317260 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.317320 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.317309 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eca8a46-979f-4e66-bcc2-d10108bf9305-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.317367 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.317335 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.349344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.349255 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:33.396869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.396820 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9"] Nov 02 06:38:33.419361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.419326 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.419554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.419507 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.419631 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.419606 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.419689 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.419634 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.419753 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.419743 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbgs2\" (UniqueName: \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-kube-api-access-cbgs2\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.419808 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.419795 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.419864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.419800 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-certs\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.419864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.419826 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.419971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.419879 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.419971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.419931 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eca8a46-979f-4e66-bcc2-d10108bf9305-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.420260 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.420238 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-credential-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.420422 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.420398 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-data\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.420781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.420733 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eca8a46-979f-4e66-bcc2-d10108bf9305-istiod-ca-cert\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.420892 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.420852 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-socket\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.422204 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.422181 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-envoy\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.423401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.423370 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-podinfo\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.474390 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.474356 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbgs2\" (UniqueName: \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-kube-api-access-cbgs2\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.474390 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.474377 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-token\") pod \"e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.574415 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.574329 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:33.592701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.592671 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:33.933762 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:33.933730 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:34.021383 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:34.021353 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9"] Nov 02 06:38:34.023830 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:34.023793 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:34.023932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:34.023856 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:34.023932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:34.023884 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:34.220662 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:34.220522 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:34.408671 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:34.408614 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:34.596258 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:34.596133 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:34.810781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:34.810735 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" event={"ID":"3eca8a46-979f-4e66-bcc2-d10108bf9305","Type":"ContainerStarted","Data":"291690b6303db10fb5ddeafb879b74bf32d29555b316fcb8a217bedb803b195c"} Nov 02 06:38:34.810952 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:34.810808 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" event={"ID":"3eca8a46-979f-4e66-bcc2-d10108bf9305","Type":"ContainerStarted","Data":"59d6d13340181a4c3e6cd98b7e7a44be4138e50bc37d058b2cab2e3bab51b85a"} Nov 02 06:38:34.974732 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:34.974682 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" podStartSLOduration=1.974667207 podStartE2EDuration="1.974667207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:34.970464793 +0000 UTC m=+1291.418841551" watchObservedRunningTime="2026-11-02 06:38:34.974667207 +0000 UTC m=+1291.423043962" Nov 02 06:38:35.603454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:35.603420 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:35.606121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:35.606091 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 06:38:35.606370 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:35.606344 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" podUID="3eca8a46-979f-4e66-bcc2-d10108bf9305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 06:38:36.314418 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.314386 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl"] Nov 02 06:38:36.318744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.318728 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.372350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.372314 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hptbf\"" Nov 02 06:38:36.401098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.398168 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.401098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.398235 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.401098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.398306 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.401098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.398344 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.401098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.398377 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgl7j\" (UniqueName: \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-kube-api-access-xgl7j\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.401098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.398441 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.401098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.398477 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.401098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.398501 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.401098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.398549 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1abd166-ac13-4582-a584-93ff1f7d8314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.423869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.423832 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl"] Nov 02 06:38:36.509219 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.509181 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.509485 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.509458 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.509647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.509532 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.509647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.509551 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.509647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.509595 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.509647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.509630 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgl7j\" (UniqueName: \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-kube-api-access-xgl7j\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.509981 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.509698 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.509981 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.509733 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.509981 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.509762 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.509981 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.509816 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1abd166-ac13-4582-a584-93ff1f7d8314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.509981 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.509818 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.510602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.510322 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.510897 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.510874 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1abd166-ac13-4582-a584-93ff1f7d8314-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.511012 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.510947 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.512540 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.512516 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.512791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.512774 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.565482 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.565396 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.565482 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.565464 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgl7j\" (UniqueName: \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-kube-api-access-xgl7j\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-wrkhl\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:36.608248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.608203 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 06:38:36.608471 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.608268 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" podUID="3eca8a46-979f-4e66-bcc2-d10108bf9305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 06:38:36.632777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:36.632744 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:37.086100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:37.084045 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl"] Nov 02 06:38:37.182468 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:37.182415 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:37.182711 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:37.182685 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:37.182850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:37.182826 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:37.599511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:37.599470 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 06:38:37.600012 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:37.599559 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" podUID="3eca8a46-979f-4e66-bcc2-d10108bf9305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 06:38:37.852048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:37.851088 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" event={"ID":"f1abd166-ac13-4582-a584-93ff1f7d8314","Type":"ContainerStarted","Data":"652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f"} Nov 02 06:38:37.852048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:37.851143 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" event={"ID":"f1abd166-ac13-4582-a584-93ff1f7d8314","Type":"ContainerStarted","Data":"d3b4efd39cf959f1ade3a4d249ba6cdb64bb8dac6949ef2336a9fa364a9bc616"} Nov 02 06:38:37.976756 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:37.976704 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" podStartSLOduration=1.97668555 podStartE2EDuration="1.97668555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:37.975232356 +0000 UTC m=+1294.423609132" watchObservedRunningTime="2026-11-02 06:38:37.97668555 +0000 UTC m=+1294.425062306" Nov 02 06:38:38.596795 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.596755 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 06:38:38.597048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.597020 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" podUID="3eca8a46-979f-4e66-bcc2-d10108bf9305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 06:38:38.637126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.636426 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:38.638393 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.638361 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 06:38:38.638514 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.638446 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 06:38:38.734101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.733124 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz"] Nov 02 06:38:38.738774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.738753 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:38.757649 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.757624 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz"] Nov 02 06:38:38.828101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.807342 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-d690ed79-serving-cert\"" Nov 02 06:38:38.867579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.867501 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl5q6\" (UniqueName: \"kubernetes.io/projected/2c9e45f8-0099-4e73-888e-53f251d22dcc-kube-api-access-hl5q6\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:38.867579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.867555 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-maas-api-tls\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:38.867770 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.867633 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-metrics-certs\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:38.867770 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.867716 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9e45f8-0099-4e73-888e-53f251d22dcc-openshift-service-ca\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:38.939940 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.939898 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:38.972050 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.970585 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl5q6\" (UniqueName: \"kubernetes.io/projected/2c9e45f8-0099-4e73-888e-53f251d22dcc-kube-api-access-hl5q6\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:38.972050 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.970660 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-maas-api-tls\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:38.972050 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.970784 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-metrics-certs\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:38.972050 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.970984 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9e45f8-0099-4e73-888e-53f251d22dcc-openshift-service-ca\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:38.972050 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:38:38.971881 5025 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-serving-cert: secret "maas-api-e2e-ait-d690ed79-serving-cert" not found Nov 02 06:38:38.972050 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:38:38.971948 5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-maas-api-tls podName:2c9e45f8-0099-4e73-888e-53f251d22dcc nodeName:}" failed. No retries permitted until 2026-11-02 06:38:39.471927236 +0000 UTC m=+1295.920303970 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-maas-api-tls") pod "maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" (UID: "2c9e45f8-0099-4e73-888e-53f251d22dcc") : secret "maas-api-e2e-ait-d690ed79-serving-cert" not found Nov 02 06:38:38.974148 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:38.974091 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9e45f8-0099-4e73-888e-53f251d22dcc-openshift-service-ca\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:39.038055 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.037766 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-metrics-certs\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:39.076371 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.060542 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl5q6\" (UniqueName: \"kubernetes.io/projected/2c9e45f8-0099-4e73-888e-53f251d22dcc-kube-api-access-hl5q6\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:39.135348 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.135252 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:39.358099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.355615 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:39.507096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.506233 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-maas-api-tls\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:39.538123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.538091 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-maas-api-tls\") pod \"maas-api-e2e-ait-d690ed79-55df646b67-c2dwz\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:39.601652 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.601614 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 06:38:39.601916 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.601888 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" podUID="3eca8a46-979f-4e66-bcc2-d10108bf9305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 06:38:39.634615 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.634583 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 06:38:39.634880 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.634851 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 06:38:39.653537 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.653511 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:39.938948 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:39.937611 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:40.129222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:40.124765 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:40.139659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:40.139627 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz"] Nov 02 06:38:40.320100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:40.319594 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:40.594435 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:40.594398 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 02 06:38:40.594734 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:40.594700 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" podUID="3eca8a46-979f-4e66-bcc2-d10108bf9305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 02 06:38:40.633497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:40.633464 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 06:38:40.633658 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:40.633524 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 06:38:40.875324 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:40.875242 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" event={"ID":"2c9e45f8-0099-4e73-888e-53f251d22dcc","Type":"ContainerStarted","Data":"78b768345f003637c29fc9badc245c24418d55094c2c6e829685d84ada6661f0"} Nov 02 06:38:41.637106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:41.635350 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 06:38:41.637106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:41.635408 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 06:38:41.964106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:41.963225 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" event={"ID":"2c9e45f8-0099-4e73-888e-53f251d22dcc","Type":"ContainerStarted","Data":"71fe04c596f1583ef05711a5ce007ac4f5a6c9ef7d2928a6464c30b9f68b953f"} Nov 02 06:38:41.965012 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:41.964938 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:42.082881 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:42.082817 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" podStartSLOduration=3.136961652 podStartE2EDuration="4.082798065s" totalImagesPullingTime="945.836413ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:42.079558919 +0000 UTC m=+1298.527935696" watchObservedRunningTime="2026-11-02 06:38:42.082798065 +0000 UTC m=+1298.531174822" Nov 02 06:38:42.419469 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:42.419437 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:42.420497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:42.420475 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:42.421993 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:42.421975 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:38:42.633683 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:42.633645 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 06:38:42.633854 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:42.633700 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 06:38:43.633575 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:43.633543 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 06:38:43.634068 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:43.633592 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 06:38:43.756101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:43.754024 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:44.633385 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:44.633346 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Nov 02 06:38:44.633559 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:44.633412 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Nov 02 06:38:45.859811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:45.859777 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:45.860379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:45.860275 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:45.861251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:45.861234 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:38:47.168610 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.168532 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj"] Nov 02 06:38:47.173044 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.173023 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.274915 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.274877 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj"] Nov 02 06:38:47.282500 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.282463 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.282679 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.282651 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.282743 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.282712 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.282909 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.282887 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c5c605-5c52-4c00-9d51-701c40c993d5-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.282968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.282929 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.283116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.283097 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.283185 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.283132 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.283270 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.283252 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfv2g\" (UniqueName: \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-kube-api-access-gfv2g\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.283336 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.283321 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.385270 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.385233 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.385519 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.385298 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.385519 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.385421 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c5c605-5c52-4c00-9d51-701c40c993d5-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.385519 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.385452 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.385702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.385560 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.385702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.385585 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.385702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.385640 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfv2g\" (UniqueName: \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-kube-api-access-gfv2g\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.385872 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.385707 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.385872 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.385754 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.387014 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.386986 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-certs\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.387500 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.387429 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.388841 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.388127 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c5c605-5c52-4c00-9d51-701c40c993d5-istiod-ca-cert\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.392102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.391529 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-credential-socket\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.392102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.392038 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-podinfo\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.393088 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.393048 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-data\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.394407 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.394383 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-envoy\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.440675 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.440615 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-token\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.446332 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.446283 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfv2g\" (UniqueName: \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-kube-api-access-gfv2g\") pod \"e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.486458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.486395 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:47.919071 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.919037 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj"] Nov 02 06:38:47.955024 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:38:47.955000 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c5c605_5c52_4c00_9d51_701c40c993d5.slice/crio-f5eca32d0b6c6ef754bebfb21d0a1650986a484092fde3e00bcdfea6a916f4d1 WatchSource:0}: Error finding container f5eca32d0b6c6ef754bebfb21d0a1650986a484092fde3e00bcdfea6a916f4d1: Status 404 returned error can't find the container with id f5eca32d0b6c6ef754bebfb21d0a1650986a484092fde3e00bcdfea6a916f4d1 Nov 02 06:38:47.957906 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.957863 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:47.957996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.957957 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:47.958036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.958000 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:38:47.972889 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.972855 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649"] Nov 02 06:38:47.978758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:47.978723 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.009299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.009272 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" event={"ID":"62c5c605-5c52-4c00-9d51-701c40c993d5","Type":"ContainerStarted","Data":"f5eca32d0b6c6ef754bebfb21d0a1650986a484092fde3e00bcdfea6a916f4d1"} Nov 02 06:38:48.028626 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.028603 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-14448910-serving-cert\"" Nov 02 06:38:48.030405 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.030379 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649"] Nov 02 06:38:48.096055 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.096019 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed06522d-f11c-4e67-a486-f5270fc2bfac-openshift-service-ca\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.096290 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.096182 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-maas-api-tls\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.096373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.096354 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-256x9\" (UniqueName: \"kubernetes.io/projected/ed06522d-f11c-4e67-a486-f5270fc2bfac-kube-api-access-256x9\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.096569 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.096542 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-metrics-certs\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.197655 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.197576 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-metrics-certs\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.198015 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.197725 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed06522d-f11c-4e67-a486-f5270fc2bfac-openshift-service-ca\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.198015 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.197768 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-maas-api-tls\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.198015 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.197801 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-256x9\" (UniqueName: \"kubernetes.io/projected/ed06522d-f11c-4e67-a486-f5270fc2bfac-kube-api-access-256x9\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.198015 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:38:48.197904 5025 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-14448910-serving-cert: secret "maas-api-e2e-derive-14448910-serving-cert" not found Nov 02 06:38:48.198015 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:38:48.197972 5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-maas-api-tls podName:ed06522d-f11c-4e67-a486-f5270fc2bfac nodeName:}" failed. No retries permitted until 2026-11-02 06:38:48.69795484 +0000 UTC m=+1305.146331576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-maas-api-tls") pod "maas-api-e2e-derive-14448910-57b67b9c79-8z649" (UID: "ed06522d-f11c-4e67-a486-f5270fc2bfac") : secret "maas-api-e2e-derive-14448910-serving-cert" not found Nov 02 06:38:48.198446 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.198427 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed06522d-f11c-4e67-a486-f5270fc2bfac-openshift-service-ca\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.200065 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.200047 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-metrics-certs\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.206615 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.206590 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:48.261197 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.261049 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-256x9\" (UniqueName: \"kubernetes.io/projected/ed06522d-f11c-4e67-a486-f5270fc2bfac-kube-api-access-256x9\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.378753 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.378718 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:48.596903 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.596786 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:48.758042 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.758006 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-maas-api-tls\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.760470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.760435 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-maas-api-tls\") pod \"maas-api-e2e-derive-14448910-57b67b9c79-8z649\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.907687 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.907353 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:48.994348 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:48.994315 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:38:49.026189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:49.026149 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" event={"ID":"62c5c605-5c52-4c00-9d51-701c40c993d5","Type":"ContainerStarted","Data":"5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676"} Nov 02 06:38:49.243835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:49.243781 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" podStartSLOduration=2.243765299 podStartE2EDuration="2.243765299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:49.239414932 +0000 UTC m=+1305.687791690" watchObservedRunningTime="2026-11-02 06:38:49.243765299 +0000 UTC m=+1305.692142054" Nov 02 06:38:49.277985 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:49.277955 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649"] Nov 02 06:38:49.278846 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:38:49.278818 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded06522d_f11c_4e67_a486_f5270fc2bfac.slice/crio-3ae13e034beb4c35a3aadb29a937af8c7e98fbd96147461f02b7f263b1ca34c3 WatchSource:0}: Error finding container 3ae13e034beb4c35a3aadb29a937af8c7e98fbd96147461f02b7f263b1ca34c3: Status 404 returned error can't find the container with id 3ae13e034beb4c35a3aadb29a937af8c7e98fbd96147461f02b7f263b1ca34c3 Nov 02 06:38:49.489766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:49.489732 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:49.490116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:49.490091 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:38:49.490300 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:49.490276 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" podUID="62c5c605-5c52-4c00-9d51-701c40c993d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 06:38:50.082842 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:50.081819 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" event={"ID":"ed06522d-f11c-4e67-a486-f5270fc2bfac","Type":"ContainerStarted","Data":"3ae13e034beb4c35a3aadb29a937af8c7e98fbd96147461f02b7f263b1ca34c3"} Nov 02 06:38:50.500103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:50.499626 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:38:50.500103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:50.499684 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" podUID="62c5c605-5c52-4c00-9d51-701c40c993d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 06:38:51.149125 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:51.139688 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" event={"ID":"ed06522d-f11c-4e67-a486-f5270fc2bfac","Type":"ContainerStarted","Data":"7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9"} Nov 02 06:38:51.149125 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:51.139780 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:38:51.280818 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:51.280765 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" podStartSLOduration=3.343525747 podStartE2EDuration="4.280750229s" totalImagesPullingTime="937.224482ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:38:51.278188678 +0000 UTC m=+1307.726565434" watchObservedRunningTime="2026-11-02 06:38:51.280750229 +0000 UTC m=+1307.729126990" Nov 02 06:38:51.490096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:51.489373 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:38:51.490096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:51.489433 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" podUID="62c5c605-5c52-4c00-9d51-701c40c993d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 06:38:52.508101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:52.505709 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:38:52.508101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:52.505772 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" podUID="62c5c605-5c52-4c00-9d51-701c40c993d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 06:38:52.936694 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:52.936645 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:53.157139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:53.153960 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:53.353411 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:53.353324 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:53.509382 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:53.509338 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:38:53.509865 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:53.509404 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" podUID="62c5c605-5c52-4c00-9d51-701c40c993d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 06:38:54.487405 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:54.487365 5025 patch_prober.go:29] interesting pod/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Nov 02 06:38:54.487609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:54.487443 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" podUID="62c5c605-5c52-4c00-9d51-701c40c993d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Nov 02 06:38:55.520942 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:55.520906 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:55.521531 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:55.521342 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:55.522119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:55.522071 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:38:57.423441 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:57.422704 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:38:58.187237 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:38:58.187210 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:39:02.217864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:02.217826 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:02.770410 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:39:02.770366 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:39:06.883837 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:06.883787 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:11.505617 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:11.505585 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:12.513350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:12.513317 5025 generic.go:356] "Generic (PLEG): container finished" podID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerID="a9de961f214c4c99e3aaae9430966f6731cced5066f5fe8b1ae53044dd216f8a" exitCode=0 Nov 02 06:39:12.513819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:12.513388 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" event={"ID":"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5","Type":"ContainerDied","Data":"a9de961f214c4c99e3aaae9430966f6731cced5066f5fe8b1ae53044dd216f8a"} Nov 02 06:39:12.513866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:12.513839 5025 scope.go:122] "RemoveContainer" containerID="a9de961f214c4c99e3aaae9430966f6731cced5066f5fe8b1ae53044dd216f8a" Nov 02 06:39:12.515504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:12.515478 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:12.515574 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:12.515546 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:12.515612 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:12.515578 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:13.522702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:13.522671 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" event={"ID":"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5","Type":"ContainerStarted","Data":"20f2cec113f4eb0bd473fcef08c5a15387843d82edc01319340c346d2d3617aa"} Nov 02 06:39:14.213466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:14.213425 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:39:14.213682 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:14.213649 5025 patch_prober.go:29] interesting pod/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 06:39:14.213741 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:14.213717 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 06:39:15.212702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:15.212664 5025 patch_prober.go:29] interesting pod/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 06:39:15.213204 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:15.212723 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 06:39:16.018108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:16.016693 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:16.213294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:16.213250 5025 patch_prober.go:29] interesting pod/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 06:39:16.213684 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:16.213316 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 06:39:17.212951 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:17.212916 5025 patch_prober.go:29] interesting pod/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 06:39:17.213136 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:17.212986 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 06:39:18.212856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:18.212800 5025 patch_prober.go:29] interesting pod/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Nov 02 06:39:18.213252 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:18.212868 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Nov 02 06:39:19.217312 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:19.217280 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:39:19.217788 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:19.217760 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:39:19.218599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:19.218577 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:39:20.561098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:20.547550 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:21.025424 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.025372 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:21.669833 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.669805 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk"] Nov 02 06:39:21.674156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.674135 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.763932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.763898 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.763932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.763937 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkcpg\" (UniqueName: \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-kube-api-access-lkcpg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.764208 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.763976 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.764208 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.764050 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.764365 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.764253 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.764365 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.764320 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.764481 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.764387 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.764481 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.764460 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.764621 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.764599 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0410350-a7d7-46a7-8365-c6ae946c1106-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.866831 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.866792 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk"] Nov 02 06:39:21.867024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.866887 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkcpg\" (UniqueName: \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-kube-api-access-lkcpg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.867024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.866956 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.867024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.866984 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.867297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.867121 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.867297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.867181 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.867297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.867213 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.867297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.867283 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.867512 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.867389 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0410350-a7d7-46a7-8365-c6ae946c1106-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.867601 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.867545 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.867805 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.867613 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.867894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.867828 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.868139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.868115 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.868823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.868800 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0410350-a7d7-46a7-8365-c6ae946c1106-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.869930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.869886 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.871365 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.871320 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.872232 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.872209 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.921825 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.921757 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkcpg\" (UniqueName: \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-kube-api-access-lkcpg\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.921825 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.921816 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ksftk\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:21.990102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:21.990052 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:22.414320 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:22.414287 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk"] Nov 02 06:39:22.414452 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:39:22.414288 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0410350_a7d7_46a7_8365_c6ae946c1106.slice/crio-e95755de3675382e4fce973936001f9ed06ef82a86cdb8ad9c56d0fc78877dd5 WatchSource:0}: Error finding container e95755de3675382e4fce973936001f9ed06ef82a86cdb8ad9c56d0fc78877dd5: Status 404 returned error can't find the container with id e95755de3675382e4fce973936001f9ed06ef82a86cdb8ad9c56d0fc78877dd5 Nov 02 06:39:22.416925 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:22.416890 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:22.417028 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:22.416982 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:22.417028 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:22.417013 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:22.611527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:22.611495 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" event={"ID":"d0410350-a7d7-46a7-8365-c6ae946c1106","Type":"ContainerStarted","Data":"4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e"} Nov 02 06:39:22.611709 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:22.611547 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" event={"ID":"d0410350-a7d7-46a7-8365-c6ae946c1106","Type":"ContainerStarted","Data":"e95755de3675382e4fce973936001f9ed06ef82a86cdb8ad9c56d0fc78877dd5"} Nov 02 06:39:22.758002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:22.757951 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" podStartSLOduration=1.757937759 podStartE2EDuration="1.757937759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:39:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:39:22.741724671 +0000 UTC m=+1339.190101445" watchObservedRunningTime="2026-11-02 06:39:22.757937759 +0000 UTC m=+1339.206314579" Nov 02 06:39:22.992206 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:22.992168 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:24.005099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:24.002134 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ksftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 06:39:24.005099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:24.002191 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 06:39:24.994101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:24.990388 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ksftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 06:39:24.994101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:24.990447 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 06:39:25.305689 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:25.305606 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:26.007100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:26.003511 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ksftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 06:39:26.007100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:26.003577 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 06:39:26.993488 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:26.993448 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ksftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 06:39:26.994160 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:26.994122 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 06:39:28.004101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:27.999173 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ksftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 06:39:28.004101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:27.999245 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 06:39:28.991146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:28.991106 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ksftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 06:39:28.991328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:28.991182 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 06:39:29.907918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:29.907878 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:29.997124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:29.994594 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ksftk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body= Nov 02 06:39:29.997124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:29.994660 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" Nov 02 06:39:30.995044 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:30.995010 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:30.995537 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:30.995413 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:30.996050 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:30.996031 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:39:32.708508 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:39:32.708471 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:39:34.617300 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:34.617266 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:36.760223 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.760185 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g"] Nov 02 06:39:36.764646 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.764626 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.859435 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.859399 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.859599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.859547 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.859657 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.859641 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.859706 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.859694 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.859767 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.859719 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.859858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.859846 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.859920 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.859906 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.860017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.859997 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.860124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.860101 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjtc\" (UniqueName: \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-kube-api-access-fhjtc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.866756 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.866724 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g"] Nov 02 06:39:36.961368 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961330 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.961548 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961402 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.961548 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961449 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.961548 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961469 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.961722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961545 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.961722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961627 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.961722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961672 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.961871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961788 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjtc\" (UniqueName: \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-kube-api-access-fhjtc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.961871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961800 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.961871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961859 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.962034 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961919 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.962034 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.961987 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.962178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.962131 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.962328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.962309 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.963860 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.963839 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:36.964296 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:36.964274 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:37.016719 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:37.016639 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:37.016843 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:37.016819 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjtc\" (UniqueName: \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-kube-api-access-fhjtc\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-pjv7g\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:37.078824 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:37.078790 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:37.504960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:37.504928 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g"] Nov 02 06:39:37.505113 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:39:37.504932 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ca03e3_b80b_498c_af13_b549f3e1ab54.slice/crio-b97d7a824f0f2cee7555f3ced1e70318af4276642ec1138a8acf5dfa9c07a59c WatchSource:0}: Error finding container b97d7a824f0f2cee7555f3ced1e70318af4276642ec1138a8acf5dfa9c07a59c: Status 404 returned error can't find the container with id b97d7a824f0f2cee7555f3ced1e70318af4276642ec1138a8acf5dfa9c07a59c Nov 02 06:39:37.507840 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:37.507809 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:37.507911 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:37.507873 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:37.507911 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:37.507901 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:39:37.725218 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:37.725185 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" event={"ID":"b6ca03e3-b80b-498c-af13-b549f3e1ab54","Type":"ContainerStarted","Data":"1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711"} Nov 02 06:39:37.725374 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:37.725239 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" event={"ID":"b6ca03e3-b80b-498c-af13-b549f3e1ab54","Type":"ContainerStarted","Data":"b97d7a824f0f2cee7555f3ced1e70318af4276642ec1138a8acf5dfa9c07a59c"} Nov 02 06:39:37.840269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:37.840163 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" podStartSLOduration=1.840145315 podStartE2EDuration="1.840145315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:39:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:39:37.837880039 +0000 UTC m=+1354.286256817" watchObservedRunningTime="2026-11-02 06:39:37.840145315 +0000 UTC m=+1354.288522072" Nov 02 06:39:38.079240 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:38.079204 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:38.081050 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:38.081019 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:38.081211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:38.081064 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 06:39:39.079906 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:39.079874 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:39.080350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:39.080250 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 06:39:39.769252 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:39.768004 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:40.079382 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:40.079300 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:40.079382 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:40.079351 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 06:39:41.079628 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:41.079589 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:41.080024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:41.079656 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 06:39:42.080014 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:42.079977 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:42.080405 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:42.080037 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 06:39:43.079779 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:43.079745 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:43.079948 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:43.079811 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 06:39:44.079517 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:44.079420 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:44.079517 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:44.079476 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 06:39:44.805002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:44.804955 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:45.080179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:45.080097 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 02 06:39:45.080179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:45.080148 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 02 06:39:45.517320 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:45.517267 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:45.714966 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:45.714925 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:46.129661 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:46.129615 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:46.130211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:46.130190 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:46.131036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:46.131012 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:39:49.944187 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:49.939054 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:52.303099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:52.298141 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv"] Nov 02 06:39:52.426291 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:52.426260 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2"] Nov 02 06:39:52.705986 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:52.705946 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:52.897202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:52.897111 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:54.793161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:54.793123 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:56.994484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:56.994443 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z"] Nov 02 06:39:57.125719 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.125671 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx"] Nov 02 06:39:57.298582 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.298471 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" podUID="5ca3fb1a-d179-4fd4-be33-d687a9d36926" containerName="payload-processing" containerID="cri-o://0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee" gracePeriod=25 Nov 02 06:39:57.395558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.395472 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:57.427579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.427549 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" podUID="c059158d-4a54-4b16-9431-e16b8d5e5772" containerName="payload-pre-processing" containerID="cri-o://b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8" gracePeriod=25 Nov 02 06:39:57.582480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.581196 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:39:57.757811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.757779 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:39:57.806754 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.806259 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ca3fb1a-d179-4fd4-be33-d687a9d36926-kube-api-access-ptjmn\" (UniqueName: \"kubernetes.io/projected/5ca3fb1a-d179-4fd4-be33-d687a9d36926-kube-api-access-ptjmn\") pod \"5ca3fb1a-d179-4fd4-be33-d687a9d36926\" (UID: \"5ca3fb1a-d179-4fd4-be33-d687a9d36926\") " Nov 02 06:39:57.806754 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.806443 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ca3fb1a-d179-4fd4-be33-d687a9d36926-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca3fb1a-d179-4fd4-be33-d687a9d36926-plugins-config-volume\") pod \"5ca3fb1a-d179-4fd4-be33-d687a9d36926\" (UID: \"5ca3fb1a-d179-4fd4-be33-d687a9d36926\") " Nov 02 06:39:57.809462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.809421 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ca3fb1a-d179-4fd4-be33-d687a9d36926-plugins-config-volume" pod "5ca3fb1a-d179-4fd4-be33-d687a9d36926" (UID: "5ca3fb1a-d179-4fd4-be33-d687a9d36926"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:39:57.816702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.816664 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ca3fb1a-d179-4fd4-be33-d687a9d36926-kube-api-access-ptjmn" pod "5ca3fb1a-d179-4fd4-be33-d687a9d36926" (UID: "5ca3fb1a-d179-4fd4-be33-d687a9d36926"). InnerVolumeSpecName "kube-api-access-ptjmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:39:57.887522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.887486 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:39:57.909123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.909092 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c059158d-4a54-4b16-9431-e16b8d5e5772-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c059158d-4a54-4b16-9431-e16b8d5e5772-plugins-config-volume\") pod \"c059158d-4a54-4b16-9431-e16b8d5e5772\" (UID: \"c059158d-4a54-4b16-9431-e16b8d5e5772\") " Nov 02 06:39:57.909406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.909389 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c059158d-4a54-4b16-9431-e16b8d5e5772-kube-api-access-m5tsj\" (UniqueName: \"kubernetes.io/projected/c059158d-4a54-4b16-9431-e16b8d5e5772-kube-api-access-m5tsj\") pod \"c059158d-4a54-4b16-9431-e16b8d5e5772\" (UID: \"c059158d-4a54-4b16-9431-e16b8d5e5772\") " Nov 02 06:39:57.909494 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.909449 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c059158d-4a54-4b16-9431-e16b8d5e5772-plugins-config-volume" pod "c059158d-4a54-4b16-9431-e16b8d5e5772" (UID: "c059158d-4a54-4b16-9431-e16b8d5e5772"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:39:57.909770 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.909753 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptjmn\" (UniqueName: \"kubernetes.io/projected/5ca3fb1a-d179-4fd4-be33-d687a9d36926-kube-api-access-ptjmn\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:57.909836 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.909769 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ca3fb1a-d179-4fd4-be33-d687a9d36926-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:57.909836 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.909780 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c059158d-4a54-4b16-9431-e16b8d5e5772-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:57.911783 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:57.911754 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c059158d-4a54-4b16-9431-e16b8d5e5772-kube-api-access-m5tsj" pod "c059158d-4a54-4b16-9431-e16b8d5e5772" (UID: "c059158d-4a54-4b16-9431-e16b8d5e5772"). InnerVolumeSpecName "kube-api-access-m5tsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:39:58.011777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.011737 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5tsj\" (UniqueName: \"kubernetes.io/projected/c059158d-4a54-4b16-9431-e16b8d5e5772-kube-api-access-m5tsj\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:39:58.105317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.105223 5025 generic.go:356] "Generic (PLEG): container finished" podID="c059158d-4a54-4b16-9431-e16b8d5e5772" containerID="b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8" exitCode=0 Nov 02 06:39:58.105317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.105273 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" Nov 02 06:39:58.105317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.105301 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" event={"ID":"c059158d-4a54-4b16-9431-e16b8d5e5772","Type":"ContainerDied","Data":"b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8"} Nov 02 06:39:58.105613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.105363 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2" event={"ID":"c059158d-4a54-4b16-9431-e16b8d5e5772","Type":"ContainerDied","Data":"d5e0f9ba1160972706e93b106677a8d13522ec7bea4caa1b7d70638294ff6271"} Nov 02 06:39:58.105613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.105391 5025 scope.go:122] "RemoveContainer" containerID="b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8" Nov 02 06:39:58.107149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.107122 5025 generic.go:356] "Generic (PLEG): container finished" podID="5ca3fb1a-d179-4fd4-be33-d687a9d36926" containerID="0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee" exitCode=0 Nov 02 06:39:58.107259 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.107183 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" Nov 02 06:39:58.107259 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.107203 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" event={"ID":"5ca3fb1a-d179-4fd4-be33-d687a9d36926","Type":"ContainerDied","Data":"0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee"} Nov 02 06:39:58.107259 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.107245 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv" event={"ID":"5ca3fb1a-d179-4fd4-be33-d687a9d36926","Type":"ContainerDied","Data":"d23cfabd0ea918831417a472a72aab6710d097a50dd1665881a5fa2d7220b6c6"} Nov 02 06:39:58.119048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.119020 5025 scope.go:122] "RemoveContainer" containerID="b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8" Nov 02 06:39:58.119650 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:39:58.119444 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8\": container with ID starting with b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8 not found: ID does not exist" containerID="b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8" Nov 02 06:39:58.119755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.119664 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8"} err="failed to get container status \"b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8\": rpc error: code = NotFound desc = could not find container \"b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8\": container with ID starting with b58730a1f42497a4408a8cbd9d122c7ecdfaaef408ca6b08535c38e092e91eb8 not found: ID does not exist" Nov 02 06:39:58.119755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.119690 5025 scope.go:122] "RemoveContainer" containerID="0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee" Nov 02 06:39:58.130868 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.130845 5025 scope.go:122] "RemoveContainer" containerID="0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee" Nov 02 06:39:58.131213 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:39:58.131192 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee\": container with ID starting with 0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee not found: ID does not exist" containerID="0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee" Nov 02 06:39:58.131291 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.131220 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee"} err="failed to get container status \"0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee\": rpc error: code = NotFound desc = could not find container \"0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee\": container with ID starting with 0a3db251e051543c675d75d3be72dfeac1b0f0fed494c6444f70ed6ad6b274ee not found: ID does not exist" Nov 02 06:39:58.264387 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.264353 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv"] Nov 02 06:39:58.271170 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.271140 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-15867003-9f55bc5f9-ftzxv"] Nov 02 06:39:58.428332 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.428266 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2"] Nov 02 06:39:58.461097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:58.453710 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-15867003-747474c7c5-5ndh2"] Nov 02 06:39:59.669833 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:59.669802 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ca3fb1a-d179-4fd4-be33-d687a9d36926" path="/var/lib/kubelet/pods/5ca3fb1a-d179-4fd4-be33-d687a9d36926/volumes" Nov 02 06:39:59.670328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:59.670262 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c059158d-4a54-4b16-9431-e16b8d5e5772" path="/var/lib/kubelet/pods/c059158d-4a54-4b16-9431-e16b8d5e5772/volumes" Nov 02 06:39:59.845932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:39:59.845895 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:01.140798 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:01.140766 5025 generic.go:356] "Generic (PLEG): container finished" podID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerID="870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af" exitCode=0 Nov 02 06:40:01.141340 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:01.140839 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" event={"ID":"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc","Type":"ContainerDied","Data":"870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af"} Nov 02 06:40:01.141422 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:01.141402 5025 scope.go:122] "RemoveContainer" containerID="870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af" Nov 02 06:40:01.143013 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:01.142980 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:01.143166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:01.143065 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:01.143166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:01.143154 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:01.912773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:01.912719 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:02.002101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:01.995467 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" podUID="7bf38d06-2b67-4c71-9924-1033ebdad32e" containerName="payload-processing" containerID="cri-o://934094c57bafdee88bd967233e73b0364b741b7adbc4eb1894fb338a7176f7d4" gracePeriod=25 Nov 02 06:40:02.127017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.126987 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" podUID="d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28" containerName="payload-pre-processing" containerID="cri-o://3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3" gracePeriod=25 Nov 02 06:40:02.150530 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.150501 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" event={"ID":"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc","Type":"ContainerStarted","Data":"275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb"} Nov 02 06:40:02.152226 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.152203 5025 generic.go:356] "Generic (PLEG): container finished" podID="7bf38d06-2b67-4c71-9924-1033ebdad32e" containerID="934094c57bafdee88bd967233e73b0364b741b7adbc4eb1894fb338a7176f7d4" exitCode=0 Nov 02 06:40:02.152323 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.152284 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" event={"ID":"7bf38d06-2b67-4c71-9924-1033ebdad32e","Type":"ContainerDied","Data":"934094c57bafdee88bd967233e73b0364b741b7adbc4eb1894fb338a7176f7d4"} Nov 02 06:40:02.359775 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.359744 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:40:02.420999 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.420967 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf38d06-2b67-4c71-9924-1033ebdad32e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf38d06-2b67-4c71-9924-1033ebdad32e-plugins-config-volume\") pod \"7bf38d06-2b67-4c71-9924-1033ebdad32e\" (UID: \"7bf38d06-2b67-4c71-9924-1033ebdad32e\") " Nov 02 06:40:02.421381 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.421355 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf38d06-2b67-4c71-9924-1033ebdad32e-kube-api-access-wftpb\" (UniqueName: \"kubernetes.io/projected/7bf38d06-2b67-4c71-9924-1033ebdad32e-kube-api-access-wftpb\") pod \"7bf38d06-2b67-4c71-9924-1033ebdad32e\" (UID: \"7bf38d06-2b67-4c71-9924-1033ebdad32e\") " Nov 02 06:40:02.422550 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.422527 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf38d06-2b67-4c71-9924-1033ebdad32e-plugins-config-volume" pod "7bf38d06-2b67-4c71-9924-1033ebdad32e" (UID: "7bf38d06-2b67-4c71-9924-1033ebdad32e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:02.431133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.430713 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf38d06-2b67-4c71-9924-1033ebdad32e-kube-api-access-wftpb" pod "7bf38d06-2b67-4c71-9924-1033ebdad32e" (UID: "7bf38d06-2b67-4c71-9924-1033ebdad32e"). InnerVolumeSpecName "kube-api-access-wftpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:02.455451 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.455424 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:40:02.523501 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.523410 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-plugins-config-volume\") pod \"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28\" (UID: \"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28\") " Nov 02 06:40:02.523672 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.523573 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-kube-api-access-qlz6v\" (UniqueName: \"kubernetes.io/projected/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-kube-api-access-qlz6v\") pod \"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28\" (UID: \"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28\") " Nov 02 06:40:02.523900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.523868 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-plugins-config-volume" pod "d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28" (UID: "d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:02.524452 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.524428 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wftpb\" (UniqueName: \"kubernetes.io/projected/7bf38d06-2b67-4c71-9924-1033ebdad32e-kube-api-access-wftpb\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:02.524552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.524456 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf38d06-2b67-4c71-9924-1033ebdad32e-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:02.524552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.524472 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:02.526485 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.526457 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-kube-api-access-qlz6v" pod "d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28" (UID: "d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28"). InnerVolumeSpecName "kube-api-access-qlz6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:02.630194 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:02.629769 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlz6v\" (UniqueName: \"kubernetes.io/projected/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28-kube-api-access-qlz6v\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:02.732836 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:40:02.732801 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:40:03.055794 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.055759 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:40:03.055983 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.055963 5025 patch_prober.go:29] interesting pod/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 06:40:03.056027 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.056013 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 06:40:03.160563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.160525 5025 generic.go:356] "Generic (PLEG): container finished" podID="d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28" containerID="3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3" exitCode=0 Nov 02 06:40:03.161067 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.160601 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" Nov 02 06:40:03.161067 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.160610 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" event={"ID":"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28","Type":"ContainerDied","Data":"3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3"} Nov 02 06:40:03.161067 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.160675 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx" event={"ID":"d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28","Type":"ContainerDied","Data":"2e0d938fad2e54268e2987978cda48bf4547a8563468f37caa5295eadee1c54a"} Nov 02 06:40:03.161067 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.160706 5025 scope.go:122] "RemoveContainer" containerID="3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3" Nov 02 06:40:03.162204 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.162180 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" event={"ID":"7bf38d06-2b67-4c71-9924-1033ebdad32e","Type":"ContainerDied","Data":"7832d2123a3a3d0c680487ce4bc796ac2c9e2489fa720fdf03a34ea27cb5c84e"} Nov 02 06:40:03.162362 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.162345 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z" Nov 02 06:40:03.171900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.171881 5025 scope.go:122] "RemoveContainer" containerID="3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3" Nov 02 06:40:03.172354 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:40:03.172230 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3\": container with ID starting with 3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3 not found: ID does not exist" containerID="3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3" Nov 02 06:40:03.172354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.172264 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3"} err="failed to get container status \"3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3\": rpc error: code = NotFound desc = could not find container \"3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3\": container with ID starting with 3ed7a9e137ac4e701698dc6ed28c527e00cf038c82b4f7f8b41a9c7887c32fe3 not found: ID does not exist" Nov 02 06:40:03.172354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.172282 5025 scope.go:122] "RemoveContainer" containerID="934094c57bafdee88bd967233e73b0364b741b7adbc4eb1894fb338a7176f7d4" Nov 02 06:40:03.318997 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.318914 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z"] Nov 02 06:40:03.366231 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.366201 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d690ed79-5f7589685f-bch4z"] Nov 02 06:40:03.470837 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.470803 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx"] Nov 02 06:40:03.473936 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.473912 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-d690ed79-65b54fd599-ttmxx"] Nov 02 06:40:03.671350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.671311 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf38d06-2b67-4c71-9924-1033ebdad32e" path="/var/lib/kubelet/pods/7bf38d06-2b67-4c71-9924-1033ebdad32e/volumes" Nov 02 06:40:03.671816 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.671802 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28" path="/var/lib/kubelet/pods/d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28/volumes" Nov 02 06:40:03.745271 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:03.745230 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds"] Nov 02 06:40:04.040414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:04.040373 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:04.057403 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:04.055384 5025 patch_prober.go:29] interesting pod/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 06:40:04.057403 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:04.055469 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 06:40:04.237168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:04.236809 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:04.469789 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:04.469756 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:05.061000 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:05.060965 5025 patch_prober.go:29] interesting pod/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 06:40:05.061303 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:05.061273 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 06:40:06.055895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:06.055861 5025 patch_prober.go:29] interesting pod/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 06:40:06.056328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:06.055909 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 06:40:06.609728 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:06.609628 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:07.055858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:07.055825 5025 patch_prober.go:29] interesting pod/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Nov 02 06:40:07.056045 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:07.055877 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Nov 02 06:40:08.002009 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:08.001978 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:08.060157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:08.060129 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:40:08.060624 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:08.060539 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:40:08.061257 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:08.061238 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:40:08.745936 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:08.745897 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" podUID="1516cf9d-8103-4605-af30-fae2e93df166" containerName="payload-pre-processing" containerID="cri-o://697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1" gracePeriod=25 Nov 02 06:40:09.038051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.038026 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:40:09.106190 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.104925 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:09.108720 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.107173 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1516cf9d-8103-4605-af30-fae2e93df166-kube-api-access-gp7nj\" (UniqueName: \"kubernetes.io/projected/1516cf9d-8103-4605-af30-fae2e93df166-kube-api-access-gp7nj\") pod \"1516cf9d-8103-4605-af30-fae2e93df166\" (UID: \"1516cf9d-8103-4605-af30-fae2e93df166\") " Nov 02 06:40:09.108720 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.107364 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1516cf9d-8103-4605-af30-fae2e93df166-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1516cf9d-8103-4605-af30-fae2e93df166-plugins-config-volume\") pod \"1516cf9d-8103-4605-af30-fae2e93df166\" (UID: \"1516cf9d-8103-4605-af30-fae2e93df166\") " Nov 02 06:40:09.116988 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.109639 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1516cf9d-8103-4605-af30-fae2e93df166-plugins-config-volume" pod "1516cf9d-8103-4605-af30-fae2e93df166" (UID: "1516cf9d-8103-4605-af30-fae2e93df166"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:09.118612 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.118564 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1516cf9d-8103-4605-af30-fae2e93df166-kube-api-access-gp7nj" pod "1516cf9d-8103-4605-af30-fae2e93df166" (UID: "1516cf9d-8103-4605-af30-fae2e93df166"). InnerVolumeSpecName "kube-api-access-gp7nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:09.218122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.218066 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp7nj\" (UniqueName: \"kubernetes.io/projected/1516cf9d-8103-4605-af30-fae2e93df166-kube-api-access-gp7nj\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:09.218355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.218335 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1516cf9d-8103-4605-af30-fae2e93df166-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:09.261685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.261587 5025 generic.go:356] "Generic (PLEG): container finished" podID="1516cf9d-8103-4605-af30-fae2e93df166" containerID="697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1" exitCode=0 Nov 02 06:40:09.261685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.261632 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" event={"ID":"1516cf9d-8103-4605-af30-fae2e93df166","Type":"ContainerDied","Data":"697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1"} Nov 02 06:40:09.261854 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.261703 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" event={"ID":"1516cf9d-8103-4605-af30-fae2e93df166","Type":"ContainerDied","Data":"2e8e5e640bcb7662b5705e92b8bbf3da1e6db1290b11940c0fd5f20e36263550"} Nov 02 06:40:09.261854 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.261739 5025 scope.go:122] "RemoveContainer" containerID="697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1" Nov 02 06:40:09.262116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.262059 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds" Nov 02 06:40:09.276158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.276137 5025 scope.go:122] "RemoveContainer" containerID="697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1" Nov 02 06:40:09.276414 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:40:09.276399 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1\": container with ID starting with 697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1 not found: ID does not exist" containerID="697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1" Nov 02 06:40:09.276466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.276423 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1"} err="failed to get container status \"697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1\": rpc error: code = NotFound desc = could not find container \"697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1\": container with ID starting with 697b557c3307de76711220c5c84312f485eb5e899df07fa5634b4408e2e01dc1 not found: ID does not exist" Nov 02 06:40:09.323786 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.323758 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4"] Nov 02 06:40:09.419808 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.419772 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds"] Nov 02 06:40:09.425426 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.425397 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-14448910-78d6669959-c4mds"] Nov 02 06:40:09.728841 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.728802 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:09.826136 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.821721 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1516cf9d-8103-4605-af30-fae2e93df166" path="/var/lib/kubelet/pods/1516cf9d-8103-4605-af30-fae2e93df166/volumes" Nov 02 06:40:09.916104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:09.915704 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:12.687159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:12.687131 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:13.815680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:13.815637 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:14.325014 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:14.324938 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" podUID="13e5782e-4b0f-4c0c-8707-c1efa3f6c756" containerName="payload-pre-processing" containerID="cri-o://8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb" gracePeriod=25 Nov 02 06:40:14.591933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:14.591861 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:40:14.746947 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:14.746907 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-kube-api-access-4hrnj\" (UniqueName: \"kubernetes.io/projected/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-kube-api-access-4hrnj\") pod \"13e5782e-4b0f-4c0c-8707-c1efa3f6c756\" (UID: \"13e5782e-4b0f-4c0c-8707-c1efa3f6c756\") " Nov 02 06:40:14.747196 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:14.747174 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-plugins-config-volume\") pod \"13e5782e-4b0f-4c0c-8707-c1efa3f6c756\" (UID: \"13e5782e-4b0f-4c0c-8707-c1efa3f6c756\") " Nov 02 06:40:14.747556 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:14.747527 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-plugins-config-volume" pod "13e5782e-4b0f-4c0c-8707-c1efa3f6c756" (UID: "13e5782e-4b0f-4c0c-8707-c1efa3f6c756"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:14.748227 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:14.748154 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:14.749364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:14.749341 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-kube-api-access-4hrnj" pod "13e5782e-4b0f-4c0c-8707-c1efa3f6c756" (UID: "13e5782e-4b0f-4c0c-8707-c1efa3f6c756"). InnerVolumeSpecName "kube-api-access-4hrnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:14.849466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:14.849386 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hrnj\" (UniqueName: \"kubernetes.io/projected/13e5782e-4b0f-4c0c-8707-c1efa3f6c756-kube-api-access-4hrnj\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:15.303985 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:15.303947 5025 generic.go:356] "Generic (PLEG): container finished" podID="13e5782e-4b0f-4c0c-8707-c1efa3f6c756" containerID="8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb" exitCode=0 Nov 02 06:40:15.304209 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:15.303996 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" Nov 02 06:40:15.304209 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:15.304031 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" event={"ID":"13e5782e-4b0f-4c0c-8707-c1efa3f6c756","Type":"ContainerDied","Data":"8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb"} Nov 02 06:40:15.304209 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:15.304106 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4" event={"ID":"13e5782e-4b0f-4c0c-8707-c1efa3f6c756","Type":"ContainerDied","Data":"b697521bd62c217f129f5f13fc055dec3d21e206322a0d88b3dce1a13347efc5"} Nov 02 06:40:15.304209 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:15.304133 5025 scope.go:122] "RemoveContainer" containerID="8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb" Nov 02 06:40:15.314428 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:15.314407 5025 scope.go:122] "RemoveContainer" containerID="8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb" Nov 02 06:40:15.314723 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:40:15.314700 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb\": container with ID starting with 8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb not found: ID does not exist" containerID="8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb" Nov 02 06:40:15.314771 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:15.314733 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb"} err="failed to get container status \"8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb\": rpc error: code = NotFound desc = could not find container \"8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb\": container with ID starting with 8c99c6925e22316e125cf92c6b5835648cc298f008b7e2d6abfe17adcc2cedcb not found: ID does not exist" Nov 02 06:40:15.463404 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:15.463369 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4"] Nov 02 06:40:15.510737 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:15.510698 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5eed1bfb-846c894695-lbrp4"] Nov 02 06:40:15.670686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:15.670655 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e5782e-4b0f-4c0c-8707-c1efa3f6c756" path="/var/lib/kubelet/pods/13e5782e-4b0f-4c0c-8707-c1efa3f6c756/volumes" Nov 02 06:40:18.392099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:18.371040 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:21.367206 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:21.367169 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:23.010104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:23.008502 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:26.992702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:26.992669 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:27.991988 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:27.991947 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz"] Nov 02 06:40:27.992784 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:27.992755 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" podUID="2c9e45f8-0099-4e73-888e-53f251d22dcc" containerName="maas-api" containerID="cri-o://71fe04c596f1583ef05711a5ce007ac4f5a6c9ef7d2928a6464c30b9f68b953f" gracePeriod=30 Nov 02 06:40:28.330140 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.326573 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:28.601409 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.594840 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:28.618104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.617795 5025 generic.go:356] "Generic (PLEG): container finished" podID="2c9e45f8-0099-4e73-888e-53f251d22dcc" containerID="71fe04c596f1583ef05711a5ce007ac4f5a6c9ef7d2928a6464c30b9f68b953f" exitCode=0 Nov 02 06:40:28.618104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.618022 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" event={"ID":"2c9e45f8-0099-4e73-888e-53f251d22dcc","Type":"ContainerDied","Data":"71fe04c596f1583ef05711a5ce007ac4f5a6c9ef7d2928a6464c30b9f68b953f"} Nov 02 06:40:28.634479 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.634445 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:40:28.816934 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.816900 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:28.846049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.833799 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-maas-api-tls\") pod \"2c9e45f8-0099-4e73-888e-53f251d22dcc\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " Nov 02 06:40:28.846049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.834087 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9e45f8-0099-4e73-888e-53f251d22dcc-kube-api-access-hl5q6\" (UniqueName: \"kubernetes.io/projected/2c9e45f8-0099-4e73-888e-53f251d22dcc-kube-api-access-hl5q6\") pod \"2c9e45f8-0099-4e73-888e-53f251d22dcc\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " Nov 02 06:40:28.846049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.834176 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9e45f8-0099-4e73-888e-53f251d22dcc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9e45f8-0099-4e73-888e-53f251d22dcc-openshift-service-ca\") pod \"2c9e45f8-0099-4e73-888e-53f251d22dcc\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " Nov 02 06:40:28.846049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.834248 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-metrics-certs\") pod \"2c9e45f8-0099-4e73-888e-53f251d22dcc\" (UID: \"2c9e45f8-0099-4e73-888e-53f251d22dcc\") " Nov 02 06:40:28.846049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.843901 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9e45f8-0099-4e73-888e-53f251d22dcc-openshift-service-ca" pod "2c9e45f8-0099-4e73-888e-53f251d22dcc" (UID: "2c9e45f8-0099-4e73-888e-53f251d22dcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:28.846862 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.846426 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c9e45f8-0099-4e73-888e-53f251d22dcc-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:28.849731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.849604 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-maas-api-tls" pod "2c9e45f8-0099-4e73-888e-53f251d22dcc" (UID: "2c9e45f8-0099-4e73-888e-53f251d22dcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:40:28.850190 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.850152 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-metrics-certs" pod "2c9e45f8-0099-4e73-888e-53f251d22dcc" (UID: "2c9e45f8-0099-4e73-888e-53f251d22dcc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:40:28.850462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.850431 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9e45f8-0099-4e73-888e-53f251d22dcc-kube-api-access-hl5q6" pod "2c9e45f8-0099-4e73-888e-53f251d22dcc" (UID: "2c9e45f8-0099-4e73-888e-53f251d22dcc"). InnerVolumeSpecName "kube-api-access-hl5q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:28.977121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.976555 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl5q6\" (UniqueName: \"kubernetes.io/projected/2c9e45f8-0099-4e73-888e-53f251d22dcc-kube-api-access-hl5q6\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:28.977121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.976597 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:28.977121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:28.976611 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c9e45f8-0099-4e73-888e-53f251d22dcc-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:29.447273 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.447240 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:29.626034 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.626001 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" Nov 02 06:40:29.626241 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.626034 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz" event={"ID":"2c9e45f8-0099-4e73-888e-53f251d22dcc","Type":"ContainerDied","Data":"78b768345f003637c29fc9badc245c24418d55094c2c6e829685d84ada6661f0"} Nov 02 06:40:29.626241 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.626121 5025 scope.go:122] "RemoveContainer" containerID="71fe04c596f1583ef05711a5ce007ac4f5a6c9ef7d2928a6464c30b9f68b953f" Nov 02 06:40:29.637606 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.637574 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:29.783056 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.782978 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz"] Nov 02 06:40:29.789780 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.789748 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-d690ed79-55df646b67-c2dwz"] Nov 02 06:40:29.866638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.866607 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:29.899464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.899431 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv"] Nov 02 06:40:29.900101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900067 5025 cpu_manager.go:414] "containerMap: removing container" podUID="13e5782e-4b0f-4c0c-8707-c1efa3f6c756" containerName="payload-pre-processing" Nov 02 06:40:29.900159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900102 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e5782e-4b0f-4c0c-8707-c1efa3f6c756" containerName="payload-pre-processing" Nov 02 06:40:29.900159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900139 5025 cpu_manager.go:414] "containerMap: removing container" podUID="1516cf9d-8103-4605-af30-fae2e93df166" containerName="payload-pre-processing" Nov 02 06:40:29.900159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900147 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1516cf9d-8103-4605-af30-fae2e93df166" containerName="payload-pre-processing" Nov 02 06:40:29.900256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900166 5025 cpu_manager.go:414] "containerMap: removing container" podUID="7bf38d06-2b67-4c71-9924-1033ebdad32e" containerName="payload-processing" Nov 02 06:40:29.900256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900172 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf38d06-2b67-4c71-9924-1033ebdad32e" containerName="payload-processing" Nov 02 06:40:29.900256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900178 5025 cpu_manager.go:414] "containerMap: removing container" podUID="2c9e45f8-0099-4e73-888e-53f251d22dcc" containerName="maas-api" Nov 02 06:40:29.900256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900182 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9e45f8-0099-4e73-888e-53f251d22dcc" containerName="maas-api" Nov 02 06:40:29.900256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900202 5025 cpu_manager.go:414] "containerMap: removing container" podUID="5ca3fb1a-d179-4fd4-be33-d687a9d36926" containerName="payload-processing" Nov 02 06:40:29.900256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900208 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ca3fb1a-d179-4fd4-be33-d687a9d36926" containerName="payload-processing" Nov 02 06:40:29.900256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900221 5025 cpu_manager.go:414] "containerMap: removing container" podUID="c059158d-4a54-4b16-9431-e16b8d5e5772" containerName="payload-pre-processing" Nov 02 06:40:29.900256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900225 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c059158d-4a54-4b16-9431-e16b8d5e5772" containerName="payload-pre-processing" Nov 02 06:40:29.900256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900241 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28" containerName="payload-pre-processing" Nov 02 06:40:29.900256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900245 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28" containerName="payload-pre-processing" Nov 02 06:40:29.900585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900344 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="1516cf9d-8103-4605-af30-fae2e93df166" containerName="payload-pre-processing" Nov 02 06:40:29.900585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900353 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf38d06-2b67-4c71-9924-1033ebdad32e" containerName="payload-processing" Nov 02 06:40:29.900585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900358 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9e45f8-0099-4e73-888e-53f251d22dcc" containerName="maas-api" Nov 02 06:40:29.900585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900369 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ca3fb1a-d179-4fd4-be33-d687a9d36926" containerName="payload-processing" Nov 02 06:40:29.900585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900378 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="c059158d-4a54-4b16-9431-e16b8d5e5772" containerName="payload-pre-processing" Nov 02 06:40:29.900585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900385 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fcd0fc-f9ae-4ce1-bdfd-4e28dd4fad28" containerName="payload-pre-processing" Nov 02 06:40:29.900585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.900392 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e5782e-4b0f-4c0c-8707-c1efa3f6c756" containerName="payload-pre-processing" Nov 02 06:40:29.905063 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:29.905044 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.020108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.010220 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-a25b83-openshift-default-dockercfg-hcgtv\"" Nov 02 06:40:30.202099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.081549 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-data\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.202099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.181251 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-podinfo\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.202099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.181314 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-envoy\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.202099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.181349 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-certs\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.202099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.181378 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-credential-socket\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.202099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.181433 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1a9eb4-dde1-488d-96ae-8b699703edab-istiod-ca-cert\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.202099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.181601 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-socket\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.202099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.181633 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5hz\" (UniqueName: \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-kube-api-access-7q5hz\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.202099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.181666 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-token\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.202099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.081614 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv"] Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.265468 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.284549 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-data\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.284624 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-podinfo\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.284672 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-envoy\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.284702 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-certs\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.284729 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-credential-socket\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.284777 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1a9eb4-dde1-488d-96ae-8b699703edab-istiod-ca-cert\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.284912 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-socket\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.284945 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5hz\" (UniqueName: \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-kube-api-access-7q5hz\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.284977 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-token\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.286011 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-data\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.288847 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1a9eb4-dde1-488d-96ae-8b699703edab-istiod-ca-cert\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.294555 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-socket\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.318157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.295314 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-credential-socket\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.360099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.359943 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v"] Nov 02 06:40:30.360296 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.360164 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" containerID="cri-o://20f2cec113f4eb0bd473fcef08c5a15387843d82edc01319340c346d2d3617aa" gracePeriod=30 Nov 02 06:40:30.409118 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.402221 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-certs\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.484955 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.482678 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:30.513886 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.511123 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-podinfo\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.530930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.528966 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5hz\" (UniqueName: \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-kube-api-access-7q5hz\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.552120 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.538473 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-envoy\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.570121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.561823 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-token\") pod \"e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:30.826415 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.826318 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:30.827135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:30.827112 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:31.050181 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:31.050150 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:31.255477 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:31.255442 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:31.377768 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:40:31.377723 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1a9eb4_dde1_488d_96ae_8b699703edab.slice/crio-e055b81d2c7731389341d3d029562345425570ed9e0a421df91dcd97c27078c2 WatchSource:0}: Error finding container e055b81d2c7731389341d3d029562345425570ed9e0a421df91dcd97c27078c2: Status 404 returned error can't find the container with id e055b81d2c7731389341d3d029562345425570ed9e0a421df91dcd97c27078c2 Nov 02 06:40:31.378234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:31.378211 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv"] Nov 02 06:40:31.379980 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:31.379949 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:31.380035 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:31.380011 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:31.380071 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:31.380038 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:31.675771 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:31.675734 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9e45f8-0099-4e73-888e-53f251d22dcc" path="/var/lib/kubelet/pods/2c9e45f8-0099-4e73-888e-53f251d22dcc/volumes" Nov 02 06:40:31.856111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:31.854228 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:31.924101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:31.923216 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" event={"ID":"1f1a9eb4-dde1-488d-96ae-8b699703edab","Type":"ContainerStarted","Data":"68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65"} Nov 02 06:40:31.924101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:31.923291 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" event={"ID":"1f1a9eb4-dde1-488d-96ae-8b699703edab","Type":"ContainerStarted","Data":"e055b81d2c7731389341d3d029562345425570ed9e0a421df91dcd97c27078c2"} Nov 02 06:40:32.153470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:32.153372 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" podStartSLOduration=3.153357625 podStartE2EDuration="3.153357625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:32.105629238 +0000 UTC m=+1408.554006013" watchObservedRunningTime="2026-11-02 06:40:32.153357625 +0000 UTC m=+1408.601734380" Nov 02 06:40:32.373108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:32.371603 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:32.709272 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:40:32.709234 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:40:32.831724 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:32.831692 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:32.837125 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:32.837065 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:32.930278 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:32.930111 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:32.931363 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:32.931344 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:40:34.220393 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.220350 5025 patch_prober.go:29] interesting pod/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:40:34.220773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.220417 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:40:34.465644 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.465611 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649"] Nov 02 06:40:34.465858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.465833 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" podUID="ed06522d-f11c-4e67-a486-f5270fc2bfac" containerName="maas-api" containerID="cri-o://7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9" gracePeriod=30 Nov 02 06:40:34.620976 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.620931 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:34.875429 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.873294 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:40:34.905285 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.905246 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-maas-api-tls\") pod \"ed06522d-f11c-4e67-a486-f5270fc2bfac\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " Nov 02 06:40:34.905453 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.905380 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed06522d-f11c-4e67-a486-f5270fc2bfac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed06522d-f11c-4e67-a486-f5270fc2bfac-openshift-service-ca\") pod \"ed06522d-f11c-4e67-a486-f5270fc2bfac\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " Nov 02 06:40:34.905514 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.905452 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-metrics-certs\") pod \"ed06522d-f11c-4e67-a486-f5270fc2bfac\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " Nov 02 06:40:34.905514 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.905490 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed06522d-f11c-4e67-a486-f5270fc2bfac-kube-api-access-256x9\" (UniqueName: \"kubernetes.io/projected/ed06522d-f11c-4e67-a486-f5270fc2bfac-kube-api-access-256x9\") pod \"ed06522d-f11c-4e67-a486-f5270fc2bfac\" (UID: \"ed06522d-f11c-4e67-a486-f5270fc2bfac\") " Nov 02 06:40:34.906008 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.905981 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed06522d-f11c-4e67-a486-f5270fc2bfac-openshift-service-ca" pod "ed06522d-f11c-4e67-a486-f5270fc2bfac" (UID: "ed06522d-f11c-4e67-a486-f5270fc2bfac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:34.906373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.906339 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed06522d-f11c-4e67-a486-f5270fc2bfac-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:34.909765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.909741 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-maas-api-tls" pod "ed06522d-f11c-4e67-a486-f5270fc2bfac" (UID: "ed06522d-f11c-4e67-a486-f5270fc2bfac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:40:34.909868 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.909776 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed06522d-f11c-4e67-a486-f5270fc2bfac-kube-api-access-256x9" pod "ed06522d-f11c-4e67-a486-f5270fc2bfac" (UID: "ed06522d-f11c-4e67-a486-f5270fc2bfac"). InnerVolumeSpecName "kube-api-access-256x9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:34.910122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.910090 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-metrics-certs" pod "ed06522d-f11c-4e67-a486-f5270fc2bfac" (UID: "ed06522d-f11c-4e67-a486-f5270fc2bfac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:40:34.953834 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.953799 5025 generic.go:356] "Generic (PLEG): container finished" podID="ed06522d-f11c-4e67-a486-f5270fc2bfac" containerID="7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9" exitCode=0 Nov 02 06:40:34.954002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.953845 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" Nov 02 06:40:34.954002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.953879 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" event={"ID":"ed06522d-f11c-4e67-a486-f5270fc2bfac","Type":"ContainerDied","Data":"7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9"} Nov 02 06:40:34.954002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.953933 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649" event={"ID":"ed06522d-f11c-4e67-a486-f5270fc2bfac","Type":"ContainerDied","Data":"3ae13e034beb4c35a3aadb29a937af8c7e98fbd96147461f02b7f263b1ca34c3"} Nov 02 06:40:34.954002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.953956 5025 scope.go:122] "RemoveContainer" containerID="7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9" Nov 02 06:40:34.966488 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.966472 5025 scope.go:122] "RemoveContainer" containerID="7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9" Nov 02 06:40:34.966769 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:40:34.966752 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9\": container with ID starting with 7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9 not found: ID does not exist" containerID="7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9" Nov 02 06:40:34.966812 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:34.966779 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9"} err="failed to get container status \"7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9\": rpc error: code = NotFound desc = could not find container \"7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9\": container with ID starting with 7531c90d8e8115aa6955a3b3893d4e162c7b16cbe6acb2700cd56a8db3d2e7a9 not found: ID does not exist" Nov 02 06:40:35.005820 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:35.005730 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:35.008056 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:35.008027 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:35.008206 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:35.008060 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed06522d-f11c-4e67-a486-f5270fc2bfac-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:35.008206 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:35.008094 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-256x9\" (UniqueName: \"kubernetes.io/projected/ed06522d-f11c-4e67-a486-f5270fc2bfac-kube-api-access-256x9\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:35.100688 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:35.100656 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649"] Nov 02 06:40:35.103779 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:35.103756 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-14448910-57b67b9c79-8z649"] Nov 02 06:40:35.216642 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:35.216605 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:35.541020 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:35.540982 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:35.844428 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:35.841895 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:35.848979 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:35.848435 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed06522d-f11c-4e67-a486-f5270fc2bfac" path="/var/lib/kubelet/pods/ed06522d-f11c-4e67-a486-f5270fc2bfac/volumes" Nov 02 06:40:36.334153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.333190 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:36.340202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.339821 5025 generic.go:356] "Generic (PLEG): container finished" podID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerID="20f2cec113f4eb0bd473fcef08c5a15387843d82edc01319340c346d2d3617aa" exitCode=0 Nov 02 06:40:36.340202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.339877 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" event={"ID":"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5","Type":"ContainerDied","Data":"20f2cec113f4eb0bd473fcef08c5a15387843d82edc01319340c346d2d3617aa"} Nov 02 06:40:36.340202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.339935 5025 scope.go:122] "RemoveContainer" containerID="a9de961f214c4c99e3aaae9430966f6731cced5066f5fe8b1ae53044dd216f8a" Nov 02 06:40:36.364927 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.364901 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:40:36.556917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.556833 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istiod-ca-cert\") pod \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " Nov 02 06:40:36.556917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.556873 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-data\") pod \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " Nov 02 06:40:36.556917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.556899 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-kube-api-access-5jx6f\" (UniqueName: \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-kube-api-access-5jx6f\") pod \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " Nov 02 06:40:36.556917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.556919 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-envoy\") pod \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " Nov 02 06:40:36.557602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.556945 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-credential-socket\") pod \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " Nov 02 06:40:36.557602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.557127 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-podinfo\") pod \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " Nov 02 06:40:36.557602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.557174 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-data" pod "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" (UID: "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:36.557602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.557235 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-certs\") pod \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " Nov 02 06:40:36.557602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.557316 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-credential-socket" pod "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" (UID: "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:36.557602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.557333 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istiod-ca-cert" pod "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" (UID: "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:36.557602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.557341 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-token\" (UniqueName: \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-token\") pod \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " Nov 02 06:40:36.557602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.557417 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-socket\") pod \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\" (UID: \"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5\") " Nov 02 06:40:36.557602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.557510 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-certs" pod "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" (UID: "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:36.558018 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.557619 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-socket" pod "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" (UID: "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:36.558244 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.558226 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.558328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.558252 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.558328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.558266 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.558328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.558279 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.558328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.558294 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.559298 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.559276 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-podinfo" pod "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" (UID: "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:40:36.559621 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.559592 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-token" pod "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" (UID: "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:36.559720 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.559672 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-envoy" pod "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" (UID: "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:36.559781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.559717 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-kube-api-access-5jx6f" pod "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" (UID: "8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5"). InnerVolumeSpecName "kube-api-access-5jx6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:36.626680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.626644 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:36.660172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.660139 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.660172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.660168 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jx6f\" (UniqueName: \"kubernetes.io/projected/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-kube-api-access-5jx6f\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.660354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.660180 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.660354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.660189 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:36.825289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.825200 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:36.900777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.900745 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58"] Nov 02 06:40:36.900971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:36.900952 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" containerID="cri-o://275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb" gracePeriod=30 Nov 02 06:40:37.127534 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.120942 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:37.233993 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.233961 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5"] Nov 02 06:40:37.237003 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.236974 5025 cpu_manager.go:414] "containerMap: removing container" podUID="ed06522d-f11c-4e67-a486-f5270fc2bfac" containerName="maas-api" Nov 02 06:40:37.237285 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.237269 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed06522d-f11c-4e67-a486-f5270fc2bfac" containerName="maas-api" Nov 02 06:40:37.237396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.237385 5025 cpu_manager.go:414] "containerMap: removing container" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" Nov 02 06:40:37.237467 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.237457 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" Nov 02 06:40:37.237586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.237576 5025 cpu_manager.go:414] "containerMap: removing container" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" Nov 02 06:40:37.237663 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.237653 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" Nov 02 06:40:37.237921 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.237906 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed06522d-f11c-4e67-a486-f5270fc2bfac" containerName="maas-api" Nov 02 06:40:37.238032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.238022 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" Nov 02 06:40:37.359811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.359775 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.374580 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.374432 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" event={"ID":"8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5","Type":"ContainerDied","Data":"488bd7df8a3ece3802e89809e7bf09f1d11e09c04a6815d21de2a89ab9ba69e0"} Nov 02 06:40:37.374867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.374846 5025 scope.go:122] "RemoveContainer" containerID="20f2cec113f4eb0bd473fcef08c5a15387843d82edc01319340c346d2d3617aa" Nov 02 06:40:37.379872 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.379796 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v" Nov 02 06:40:37.383141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.383067 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5"] Nov 02 06:40:37.405722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.405686 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b838f9ff-708c-46f5-adec-d4425d46c695-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.405722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.405718 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.405882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.405740 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.405882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.405765 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b838f9ff-708c-46f5-adec-d4425d46c695-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.405968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.405952 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.406120 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.406071 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.406238 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.406196 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.406307 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.406288 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78wbh\" (UniqueName: \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-kube-api-access-78wbh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.406357 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.406348 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.507843 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.507806 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.507994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.507888 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78wbh\" (UniqueName: \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-kube-api-access-78wbh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.507994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.507936 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.507994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.507986 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b838f9ff-708c-46f5-adec-d4425d46c695-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.508214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.508003 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.508214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.508026 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.508214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.508049 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b838f9ff-708c-46f5-adec-d4425d46c695-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.508214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.508163 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.508214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.508201 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.508440 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.508301 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.508440 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.508402 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.508564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.508455 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.508668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.508644 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.508803 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.508782 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b838f9ff-708c-46f5-adec-d4425d46c695-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.510412 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.510393 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.510662 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.510644 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b838f9ff-708c-46f5-adec-d4425d46c695-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.562969 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.562935 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.563435 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.562995 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78wbh\" (UniqueName: \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-kube-api-access-78wbh\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-tlwz5\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.643143 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.643039 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v"] Nov 02 06:40:37.651106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.651060 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-d690ed79-openshift-default-cd759946f-nrk9v"] Nov 02 06:40:37.670505 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.670472 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" path="/var/lib/kubelet/pods/8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5/volumes" Nov 02 06:40:37.693062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.693034 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:37.789101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.778180 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:37.964112 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:37.964070 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:38.062575 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.062543 5025 patch_prober.go:29] interesting pod/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:40:38.062747 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.062609 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:40:38.170263 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.170237 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5"] Nov 02 06:40:38.171021 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:40:38.170996 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb838f9ff_708c_46f5_adec_d4425d46c695.slice/crio-ec1362123961f517895c0f89255caac5591d72a4d8c94cf80db1449f9c8ec3b7 WatchSource:0}: Error finding container ec1362123961f517895c0f89255caac5591d72a4d8c94cf80db1449f9c8ec3b7: Status 404 returned error can't find the container with id ec1362123961f517895c0f89255caac5591d72a4d8c94cf80db1449f9c8ec3b7 Nov 02 06:40:38.173744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.173709 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:38.173819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.173781 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:38.173819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.173810 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:40:38.382573 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.382534 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" event={"ID":"b838f9ff-708c-46f5-adec-d4425d46c695","Type":"ContainerStarted","Data":"9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0"} Nov 02 06:40:38.382777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.382603 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" event={"ID":"b838f9ff-708c-46f5-adec-d4425d46c695","Type":"ContainerStarted","Data":"ec1362123961f517895c0f89255caac5591d72a4d8c94cf80db1449f9c8ec3b7"} Nov 02 06:40:38.503478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.503374 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" podStartSLOduration=1.503355992 podStartE2EDuration="1.503355992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:38.497788453 +0000 UTC m=+1414.946165234" watchObservedRunningTime="2026-11-02 06:40:38.503355992 +0000 UTC m=+1414.951732748" Nov 02 06:40:38.605864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.605835 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:38.693972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.693938 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:38.873426 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:38.873342 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:39.146155 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.143152 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:39.460155 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.429851 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:39.526607 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.492224 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5"] Nov 02 06:40:39.526607 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.493645 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4a4872-6bcb-4c47-b62c-7ddab59ce9e5" containerName="istio-proxy" Nov 02 06:40:39.549095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.542440 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:40:39.722146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.717513 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-a25b83\"" Nov 02 06:40:39.780296 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.780259 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-a25b83-dockercfg-8mf6w\"" Nov 02 06:40:39.817543 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.817508 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:39.817731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.817564 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 06:40:39.822814 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.822786 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5"] Nov 02 06:40:39.917012 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.916967 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb84f\" (UniqueName: \"kubernetes.io/projected/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-kube-api-access-mb84f\") pod \"payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5\" (UID: \"3b5b8fd5-abb5-439a-90f1-a07a4f9de230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:40:39.917365 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:39.917331 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5\" (UID: \"3b5b8fd5-abb5-439a-90f1-a07a4f9de230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:40:40.019098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:40.018976 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5\" (UID: \"3b5b8fd5-abb5-439a-90f1-a07a4f9de230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:40:40.019098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:40.019064 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb84f\" (UniqueName: \"kubernetes.io/projected/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-kube-api-access-mb84f\") pod \"payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5\" (UID: \"3b5b8fd5-abb5-439a-90f1-a07a4f9de230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:40:40.019759 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:40.019736 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5\" (UID: \"3b5b8fd5-abb5-439a-90f1-a07a4f9de230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:40:40.074384 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:40.074354 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb84f\" (UniqueName: \"kubernetes.io/projected/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-kube-api-access-mb84f\") pod \"payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5\" (UID: \"3b5b8fd5-abb5-439a-90f1-a07a4f9de230\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:40:40.120358 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:40.120327 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:40:40.515220 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:40.514486 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5"] Nov 02 06:40:40.515921 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:40:40.515891 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5b8fd5_abb5_439a_90f1_a07a4f9de230.slice/crio-12ec92692c798d542c092bf1aa175b2c5d5e3dadc2020629bbfdb9662dc64329 WatchSource:0}: Error finding container 12ec92692c798d542c092bf1aa175b2c5d5e3dadc2020629bbfdb9662dc64329: Status 404 returned error can't find the container with id 12ec92692c798d542c092bf1aa175b2c5d5e3dadc2020629bbfdb9662dc64329 Nov 02 06:40:40.635747 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:40.635711 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" event={"ID":"3b5b8fd5-abb5-439a-90f1-a07a4f9de230","Type":"ContainerStarted","Data":"12ec92692c798d542c092bf1aa175b2c5d5e3dadc2020629bbfdb9662dc64329"} Nov 02 06:40:40.635960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:40.635943 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:40:40.707101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:40.697162 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:40.707101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:40.697220 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 06:40:40.755119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:40.755002 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" podStartSLOduration=1.754988539 podStartE2EDuration="1.754988539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:40.75195086 +0000 UTC m=+1417.200327617" watchObservedRunningTime="2026-11-02 06:40:40.754988539 +0000 UTC m=+1417.203365314" Nov 02 06:40:41.209177 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:41.209038 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:41.643796 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:41.643692 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" event={"ID":"3b5b8fd5-abb5-439a-90f1-a07a4f9de230","Type":"ContainerStarted","Data":"1147a78049020220f0ebd3be5eed1a9802418a92dbadd8e44ed375fbb2ca25e2"} Nov 02 06:40:41.707101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:41.694466 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:41.707101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:41.694527 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 06:40:42.058018 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.013839 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:42.515659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.515628 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:40:42.615109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615056 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-envoy\") pod \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " Nov 02 06:40:42.615283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615202 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-credential-socket\") pod \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " Nov 02 06:40:42.615283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615269 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istiod-ca-cert\") pod \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " Nov 02 06:40:42.615378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615355 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-socket\") pod \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " Nov 02 06:40:42.615466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615447 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-podinfo\") pod \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " Nov 02 06:40:42.615564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615495 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-certs\") pod \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " Nov 02 06:40:42.615564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615523 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-credential-socket" pod "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" (UID: "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:42.615687 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615577 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-token\" (UniqueName: \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-token\") pod \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " Nov 02 06:40:42.615687 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615634 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-data\") pod \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " Nov 02 06:40:42.615687 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615682 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-kube-api-access-dzdjl\" (UniqueName: \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-kube-api-access-dzdjl\") pod \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\" (UID: \"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc\") " Nov 02 06:40:42.615847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615702 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-socket" pod "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" (UID: "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:42.615847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615735 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istiod-ca-cert" pod "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" (UID: "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:40:42.615847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615743 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-certs" pod "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" (UID: "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:42.616003 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.615978 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-data" pod "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" (UID: "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:42.616272 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.616254 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:42.616378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.616276 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:42.616378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.616290 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:42.616378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.616303 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:42.616378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.616317 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:42.618134 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.618107 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-envoy" pod "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" (UID: "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:40:42.618291 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.618200 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-podinfo" pod "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" (UID: "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:40:42.618539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.618518 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-token" pod "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" (UID: "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:42.618608 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.618548 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-kube-api-access-dzdjl" pod "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" (UID: "45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc"). InnerVolumeSpecName "kube-api-access-dzdjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:40:42.725150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.720546 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:42.725150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.720589 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:42.725150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.720605 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:42.725150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.720619 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzdjl\" (UniqueName: \"kubernetes.io/projected/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc-kube-api-access-dzdjl\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:40:42.734107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.732591 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:42.747832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.747245 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:42.747832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.747308 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 06:40:42.766272 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.766233 5025 generic.go:356] "Generic (PLEG): container finished" podID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerID="275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb" exitCode=0 Nov 02 06:40:42.766454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.766288 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" event={"ID":"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc","Type":"ContainerDied","Data":"275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb"} Nov 02 06:40:42.766454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.766333 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" Nov 02 06:40:42.766454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.766368 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58" event={"ID":"45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc","Type":"ContainerDied","Data":"f1039338865c449bada0df3d28e6e835f61f1694fd8926b87e50e7bbf50df934"} Nov 02 06:40:42.766454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.766408 5025 scope.go:122] "RemoveContainer" containerID="275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb" Nov 02 06:40:42.778601 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.778583 5025 scope.go:122] "RemoveContainer" containerID="870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af" Nov 02 06:40:42.816751 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.816340 5025 scope.go:122] "RemoveContainer" containerID="275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb" Nov 02 06:40:42.816751 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:40:42.816699 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb\": container with ID starting with 275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb not found: ID does not exist" containerID="275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb" Nov 02 06:40:42.816751 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.816740 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb"} err="failed to get container status \"275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb\": rpc error: code = NotFound desc = could not find container \"275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb\": container with ID starting with 275ad302ffd542db2fd2195b44879a65dea5707a5b84427e5861763f781b7bdb not found: ID does not exist" Nov 02 06:40:42.817019 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.816765 5025 scope.go:122] "RemoveContainer" containerID="870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af" Nov 02 06:40:42.817637 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:40:42.817606 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af\": container with ID starting with 870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af not found: ID does not exist" containerID="870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af" Nov 02 06:40:42.817700 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.817648 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af"} err="failed to get container status \"870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af\": rpc error: code = NotFound desc = could not find container \"870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af\": container with ID starting with 870c8b2ede17a82312ed8aa7fe2806796c50d81e00909f94418d9bd9508d67af not found: ID does not exist" Nov 02 06:40:42.913336 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.913301 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:42.924844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.924816 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58"] Nov 02 06:40:42.930043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:42.930021 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-14448910-openshift-default-76b75c7cdf-2wx58"] Nov 02 06:40:43.110178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:43.106602 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:43.674013 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:43.673983 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" path="/var/lib/kubelet/pods/45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc/volumes" Nov 02 06:40:43.697508 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:43.697471 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:43.701109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:43.697760 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 06:40:43.722094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:43.722056 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:43.970969 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:43.970872 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:44.111684 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.110826 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h"] Nov 02 06:40:44.112182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.112092 5025 cpu_manager.go:414] "containerMap: removing container" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" Nov 02 06:40:44.112182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.112113 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" Nov 02 06:40:44.112182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.112174 5025 cpu_manager.go:414] "containerMap: removing container" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" Nov 02 06:40:44.112182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.112181 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" Nov 02 06:40:44.112386 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.112335 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" Nov 02 06:40:44.223383 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.223299 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h"] Nov 02 06:40:44.223676 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.223658 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:40:44.302103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.297566 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fc9fdd1c\"" Nov 02 06:40:44.390116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.368738 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fc9fdd1c-dockercfg-rlljz\"" Nov 02 06:40:44.390116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.379480 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d743ff-c4ed-4e97-9236-e337fdc88f06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h\" (UID: \"88d743ff-c4ed-4e97-9236-e337fdc88f06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:40:44.390116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.379604 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx88c\" (UniqueName: \"kubernetes.io/projected/88d743ff-c4ed-4e97-9236-e337fdc88f06-kube-api-access-wx88c\") pod \"payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h\" (UID: \"88d743ff-c4ed-4e97-9236-e337fdc88f06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:40:44.454099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.448134 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb"] Nov 02 06:40:44.454099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.449361 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="45d84c9a-d57a-4d3b-bdf0-5b9272c2a6dc" containerName="istio-proxy" Nov 02 06:40:44.475152 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.473298 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb"] Nov 02 06:40:44.475152 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.473436 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:40:44.493104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.492505 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cphm\" (UniqueName: \"kubernetes.io/projected/04fc5642-8c5b-439b-ad74-a970672a1acc-kube-api-access-8cphm\") pod \"payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb\" (UID: \"04fc5642-8c5b-439b-ad74-a970672a1acc\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:40:44.493104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.492705 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d743ff-c4ed-4e97-9236-e337fdc88f06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h\" (UID: \"88d743ff-c4ed-4e97-9236-e337fdc88f06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:40:44.493104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.492815 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fc5642-8c5b-439b-ad74-a970672a1acc-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb\" (UID: \"04fc5642-8c5b-439b-ad74-a970672a1acc\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:40:44.493104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.492844 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx88c\" (UniqueName: \"kubernetes.io/projected/88d743ff-c4ed-4e97-9236-e337fdc88f06-kube-api-access-wx88c\") pod \"payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h\" (UID: \"88d743ff-c4ed-4e97-9236-e337fdc88f06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:40:44.505140 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.494374 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d743ff-c4ed-4e97-9236-e337fdc88f06-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h\" (UID: \"88d743ff-c4ed-4e97-9236-e337fdc88f06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:40:44.606181 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.603553 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cphm\" (UniqueName: \"kubernetes.io/projected/04fc5642-8c5b-439b-ad74-a970672a1acc-kube-api-access-8cphm\") pod \"payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb\" (UID: \"04fc5642-8c5b-439b-ad74-a970672a1acc\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:40:44.606181 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.603804 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fc5642-8c5b-439b-ad74-a970672a1acc-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb\" (UID: \"04fc5642-8c5b-439b-ad74-a970672a1acc\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:40:44.646047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.610965 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fc5642-8c5b-439b-ad74-a970672a1acc-plugins-config-volume\") pod \"payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb\" (UID: \"04fc5642-8c5b-439b-ad74-a970672a1acc\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:40:44.749349 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.718275 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:44.749349 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.718378 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 06:40:44.765319 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.765242 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx88c\" (UniqueName: \"kubernetes.io/projected/88d743ff-c4ed-4e97-9236-e337fdc88f06-kube-api-access-wx88c\") pod \"payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h\" (UID: \"88d743ff-c4ed-4e97-9236-e337fdc88f06\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:40:44.805201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.804877 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cphm\" (UniqueName: \"kubernetes.io/projected/04fc5642-8c5b-439b-ad74-a970672a1acc-kube-api-access-8cphm\") pod \"payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb\" (UID: \"04fc5642-8c5b-439b-ad74-a970672a1acc\") " pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:40:44.824428 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.824395 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:40:44.851706 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:44.851676 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:40:45.231143 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:45.231110 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb"] Nov 02 06:40:45.231594 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:40:45.231564 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04fc5642_8c5b_439b_ad74_a970672a1acc.slice/crio-7c84b85b18d261a52e4794c430e7ab5cefd08f9ca1cabc0df5f470b66fdf976f WatchSource:0}: Error finding container 7c84b85b18d261a52e4794c430e7ab5cefd08f9ca1cabc0df5f470b66fdf976f: Status 404 returned error can't find the container with id 7c84b85b18d261a52e4794c430e7ab5cefd08f9ca1cabc0df5f470b66fdf976f Nov 02 06:40:45.277445 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:45.277416 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h"] Nov 02 06:40:45.284398 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:40:45.284370 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d743ff_c4ed_4e97_9236_e337fdc88f06.slice/crio-a25d67d0df4ebec2bd3b332fb2b1ff78155b380f6fb620a37da62cbd644e11b6 WatchSource:0}: Error finding container a25d67d0df4ebec2bd3b332fb2b1ff78155b380f6fb620a37da62cbd644e11b6: Status 404 returned error can't find the container with id a25d67d0df4ebec2bd3b332fb2b1ff78155b380f6fb620a37da62cbd644e11b6 Nov 02 06:40:45.695182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:45.695144 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:45.695362 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:45.695201 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 06:40:45.998559 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:45.998465 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" event={"ID":"88d743ff-c4ed-4e97-9236-e337fdc88f06","Type":"ContainerStarted","Data":"eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993"} Nov 02 06:40:45.998559 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:45.998521 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" event={"ID":"88d743ff-c4ed-4e97-9236-e337fdc88f06","Type":"ContainerStarted","Data":"a25d67d0df4ebec2bd3b332fb2b1ff78155b380f6fb620a37da62cbd644e11b6"} Nov 02 06:40:45.998802 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:45.998656 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:40:46.000315 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:46.000288 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" event={"ID":"04fc5642-8c5b-439b-ad74-a970672a1acc","Type":"ContainerStarted","Data":"8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08"} Nov 02 06:40:46.000439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:46.000333 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" event={"ID":"04fc5642-8c5b-439b-ad74-a970672a1acc","Type":"ContainerStarted","Data":"7c84b85b18d261a52e4794c430e7ab5cefd08f9ca1cabc0df5f470b66fdf976f"} Nov 02 06:40:46.000439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:46.000397 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:40:46.109257 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:46.109191 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" podStartSLOduration=2.109177968 podStartE2EDuration="2.109177968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:46.107588739 +0000 UTC m=+1422.555965520" watchObservedRunningTime="2026-11-02 06:40:46.109177968 +0000 UTC m=+1422.557554725" Nov 02 06:40:46.246907 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:46.246825 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" podStartSLOduration=2.246801268 podStartE2EDuration="2.246801268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:46.232944513 +0000 UTC m=+1422.681321278" watchObservedRunningTime="2026-11-02 06:40:46.246801268 +0000 UTC m=+1422.695178028" Nov 02 06:40:46.459786 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:46.459749 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:46.694306 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:46.694269 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Nov 02 06:40:46.694454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:46.694335 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Nov 02 06:40:46.731996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:46.731881 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:47.735223 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:47.735182 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:47.735665 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:47.735586 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:47.737364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:47.737336 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:40:48.351210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:48.351172 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:48.702273 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:48.702237 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:49.575495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:49.575451 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:49.784106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:49.783728 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:50.211351 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.197350 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p"] Nov 02 06:40:50.223174 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.223135 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p"] Nov 02 06:40:50.223644 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.223624 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:40:50.307101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.279767 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3bf02ebf\"" Nov 02 06:40:50.406103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.309005 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3bf02ebf-dockercfg-wcdgb\"" Nov 02 06:40:50.406500 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.359629 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krjhq\" (UniqueName: \"kubernetes.io/projected/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-kube-api-access-krjhq\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p\" (UID: \"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:40:50.407104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.407054 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-plugins-config-volume\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p\" (UID: \"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:40:50.521437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.521346 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krjhq\" (UniqueName: \"kubernetes.io/projected/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-kube-api-access-krjhq\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p\" (UID: \"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:40:50.521575 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.521545 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-plugins-config-volume\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p\" (UID: \"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:40:50.522279 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.522258 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-plugins-config-volume\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p\" (UID: \"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:40:50.576235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.576206 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krjhq\" (UniqueName: \"kubernetes.io/projected/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-kube-api-access-krjhq\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p\" (UID: \"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:40:50.870892 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:50.870805 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:40:51.306159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:51.306132 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p"] Nov 02 06:40:52.272109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:52.272036 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" event={"ID":"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1","Type":"ContainerStarted","Data":"72a66ff18f7efe1f77e2410eeb3d0fb2fc95c828ebf4c651217b483fb6648de6"} Nov 02 06:40:52.272882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:52.272124 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" event={"ID":"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1","Type":"ContainerStarted","Data":"d70ee041a6ba9fea8e2bb0d02d798046d866e6c93585a32bdc74b2d9292ec7c9"} Nov 02 06:40:52.272882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:52.272157 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:40:52.384124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:52.383804 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" podStartSLOduration=2.383783487 podStartE2EDuration="2.383783487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:40:52.381531664 +0000 UTC m=+1428.829908421" watchObservedRunningTime="2026-11-02 06:40:52.383783487 +0000 UTC m=+1428.832160238" Nov 02 06:40:52.831098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:52.824336 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:40:53.526099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:53.525960 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:40:53.801168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:53.780824 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:40:57.011606 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:57.011565 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:40:57.012069 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:57.011676 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:40:58.282865 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:40:58.282809 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:02.709461 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:41:02.709419 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:41:02.894031 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:02.883535 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:03.338092 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:03.338050 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:41:07.504274 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:07.504233 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:12.104100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:12.098138 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:16.369749 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.369667 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28"] Nov 02 06:41:16.375788 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.375765 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.428055 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.428023 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28"] Nov 02 06:41:16.471032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.471001 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-a25b83-serving-cert\"" Nov 02 06:41:16.491016 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.490976 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.491277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.491254 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2fc06f2-607e-44dc-beac-0048d272c8aa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.491449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.491424 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79pp\" (UniqueName: \"kubernetes.io/projected/d2fc06f2-607e-44dc-beac-0048d272c8aa-kube-api-access-d79pp\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.491640 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.491617 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.592924 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.592891 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.593116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.592966 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2fc06f2-607e-44dc-beac-0048d272c8aa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.593116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.592988 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79pp\" (UniqueName: \"kubernetes.io/projected/d2fc06f2-607e-44dc-beac-0048d272c8aa-kube-api-access-d79pp\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.593116 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:41:16.593044 5025 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-serving-cert: secret "maas-api-e2e-shared-a-w0-a25b83-serving-cert" not found Nov 02 06:41:16.593245 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:41:16.593130 5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-maas-api-tls podName:d2fc06f2-607e-44dc-beac-0048d272c8aa nodeName:}" failed. No retries permitted until 2026-11-02 06:41:17.093110129 +0000 UTC m=+1453.541486864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-maas-api-tls") pod "maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" (UID: "d2fc06f2-607e-44dc-beac-0048d272c8aa") : secret "maas-api-e2e-shared-a-w0-a25b83-serving-cert" not found Nov 02 06:41:16.593245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.593217 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.593704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.593687 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2fc06f2-607e-44dc-beac-0048d272c8aa-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.595874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.595845 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.625954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.625855 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:16.651251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.651011 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79pp\" (UniqueName: \"kubernetes.io/projected/d2fc06f2-607e-44dc-beac-0048d272c8aa-kube-api-access-d79pp\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:16.820098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:16.819620 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:17.118434 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:17.118402 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:17.120832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:17.120807 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:17.289478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:17.289449 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:17.676050 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:41:17.676018 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fc06f2_607e_44dc_beac_0048d272c8aa.slice/crio-11172f809ca89149624bc18774a4f507afc5ec31aa64fffa90d2c210182d33c2 WatchSource:0}: Error finding container 11172f809ca89149624bc18774a4f507afc5ec31aa64fffa90d2c210182d33c2: Status 404 returned error can't find the container with id 11172f809ca89149624bc18774a4f507afc5ec31aa64fffa90d2c210182d33c2 Nov 02 06:41:17.681573 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:17.681548 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28"] Nov 02 06:41:18.529380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:18.529344 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" event={"ID":"d2fc06f2-607e-44dc-beac-0048d272c8aa","Type":"ContainerStarted","Data":"11172f809ca89149624bc18774a4f507afc5ec31aa64fffa90d2c210182d33c2"} Nov 02 06:41:19.538837 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:19.538800 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" event={"ID":"d2fc06f2-607e-44dc-beac-0048d272c8aa","Type":"ContainerStarted","Data":"e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334"} Nov 02 06:41:19.539358 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:19.538938 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:19.653178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:19.653116 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" podStartSLOduration=2.673295107 podStartE2EDuration="3.653097736s" totalImagesPullingTime="979.802629ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:41:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:41:19.649963681 +0000 UTC m=+1456.098340438" watchObservedRunningTime="2026-11-02 06:41:19.653097736 +0000 UTC m=+1456.101474492" Nov 02 06:41:21.089913 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.089869 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244"] Nov 02 06:41:21.094533 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.094512 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.147404 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.147370 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244"] Nov 02 06:41:21.190594 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.190555 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fc9fdd1c-serving-cert\"" Nov 02 06:41:21.212526 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.212490 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-openshift-service-ca\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.212695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.212677 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzcxw\" (UniqueName: \"kubernetes.io/projected/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-kube-api-access-rzcxw\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.212737 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.212723 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-maas-api-tls\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.212898 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.212881 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-metrics-certs\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.314736 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.314702 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-metrics-certs\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.314930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.314802 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-openshift-service-ca\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.315007 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.314947 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzcxw\" (UniqueName: \"kubernetes.io/projected/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-kube-api-access-rzcxw\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.315070 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.315002 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-maas-api-tls\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.315203 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:41:21.315175 5025 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-serving-cert: secret "maas-api-e2e-ait-fc9fdd1c-serving-cert" not found Nov 02 06:41:21.315322 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:41:21.315250 5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-maas-api-tls podName:ee74d7a3-a68f-44a8-bb32-8bd61cdefe62 nodeName:}" failed. No retries permitted until 2026-11-02 06:41:21.815230445 +0000 UTC m=+1458.263607198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-maas-api-tls") pod "maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" (UID: "ee74d7a3-a68f-44a8-bb32-8bd61cdefe62") : secret "maas-api-e2e-ait-fc9fdd1c-serving-cert" not found Nov 02 06:41:21.315521 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.315501 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-openshift-service-ca\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.317152 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.317127 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-metrics-certs\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.374182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.371550 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:21.389866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.389835 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzcxw\" (UniqueName: \"kubernetes.io/projected/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-kube-api-access-rzcxw\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.589593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.587647 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:21.772693 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.772660 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:21.825702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.825669 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-maas-api-tls\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:21.828051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:21.828032 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-maas-api-tls\") pod \"maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:22.008356 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:22.008324 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:22.377358 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:22.377324 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244"] Nov 02 06:41:22.377795 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:41:22.377612 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee74d7a3_a68f_44a8_bb32_8bd61cdefe62.slice/crio-9c5ba69bd3d390f6004ff4810d9566736852f0065330d7851aa6099136f65d83 WatchSource:0}: Error finding container 9c5ba69bd3d390f6004ff4810d9566736852f0065330d7851aa6099136f65d83: Status 404 returned error can't find the container with id 9c5ba69bd3d390f6004ff4810d9566736852f0065330d7851aa6099136f65d83 Nov 02 06:41:22.559730 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:22.559694 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" event={"ID":"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62","Type":"ContainerStarted","Data":"9c5ba69bd3d390f6004ff4810d9566736852f0065330d7851aa6099136f65d83"} Nov 02 06:41:23.567716 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:23.567620 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" event={"ID":"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62","Type":"ContainerStarted","Data":"70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb"} Nov 02 06:41:23.567716 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:23.567695 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:23.678801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:23.678747 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" podStartSLOduration=1.818831518 podStartE2EDuration="2.67873386s" totalImagesPullingTime="859.902342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:41:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:41:23.677098759 +0000 UTC m=+1460.125475513" watchObservedRunningTime="2026-11-02 06:41:23.67873386 +0000 UTC m=+1460.127110616" Nov 02 06:41:25.550756 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:25.550724 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:41:25.717152 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:25.717119 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:26.202889 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:26.202855 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:27.411971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:27.411940 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:29.581586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:29.581549 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:41:30.825104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:30.824983 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:31.805854 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:31.805824 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:32.364204 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.364168 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:32.708897 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:41:32.708860 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:41:32.791403 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.791374 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv"] Nov 02 06:41:32.796025 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.796005 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:32.848121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.845848 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-3e96be-openshift-default-dockercfg-k68mm\"" Nov 02 06:41:32.854960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.854912 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-certs\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:32.855151 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.855049 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-token\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:32.855371 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.855344 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2zzb\" (UniqueName: \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-kube-api-access-z2zzb\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:32.855470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.855395 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-data\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:32.855470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.855430 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-envoy\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:32.855578 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.855512 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-podinfo\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:32.855644 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.855610 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-credential-socket\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:32.855703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.855649 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abe717d-5020-42af-9ad7-ab8c166f6936-istiod-ca-cert\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:32.855703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.855678 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-socket\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:32.898800 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.898764 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv"] Nov 02 06:41:33.001149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.998349 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-credential-socket\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.001149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.998415 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abe717d-5020-42af-9ad7-ab8c166f6936-istiod-ca-cert\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.001149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.998446 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-socket\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.001149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.998576 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-certs\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.001149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.998640 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-token\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.001149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.998840 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2zzb\" (UniqueName: \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-kube-api-access-z2zzb\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.001149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.998880 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-data\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.001149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.998914 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-envoy\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.001149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:32.998944 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-podinfo\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.047111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.002720 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-certs\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.047111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.003243 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-data\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.047111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.043141 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-credential-socket\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.047111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.046188 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abe717d-5020-42af-9ad7-ab8c166f6936-istiod-ca-cert\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.086848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.070960 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-socket\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.170168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.127827 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:33.170168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.147770 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-podinfo\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.199104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.193906 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-envoy\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.229815 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.208703 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-token\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.232905 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.232875 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2zzb\" (UniqueName: \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-kube-api-access-z2zzb\") pod \"e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.330088 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.329651 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:33.415766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.415446 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:33.844620 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:41:33.844588 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abe717d_5020_42af_9ad7_ab8c166f6936.slice/crio-3997780c9f8f3ba1965de263ea65429369415a2b8e822afad50f29a4f8eba360 WatchSource:0}: Error finding container 3997780c9f8f3ba1965de263ea65429369415a2b8e822afad50f29a4f8eba360: Status 404 returned error can't find the container with id 3997780c9f8f3ba1965de263ea65429369415a2b8e822afad50f29a4f8eba360 Nov 02 06:41:33.846400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.846305 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv"] Nov 02 06:41:33.847701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.847664 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:33.847781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.847734 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:33.847781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:33.847764 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:34.644480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:34.644440 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" event={"ID":"9abe717d-5020-42af-9ad7-ab8c166f6936","Type":"ContainerStarted","Data":"5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218"} Nov 02 06:41:34.644866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:34.644505 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" event={"ID":"9abe717d-5020-42af-9ad7-ab8c166f6936","Type":"ContainerStarted","Data":"3997780c9f8f3ba1965de263ea65429369415a2b8e822afad50f29a4f8eba360"} Nov 02 06:41:34.807664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:34.807613 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" podStartSLOduration=2.807597834 podStartE2EDuration="2.807597834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:41:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:41:34.807200542 +0000 UTC m=+1471.255577334" watchObservedRunningTime="2026-11-02 06:41:34.807597834 +0000 UTC m=+1471.255974590" Nov 02 06:41:35.445100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:35.443211 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:35.464716 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:35.464676 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:35.467923 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:35.467899 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:35.687724 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:35.687690 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:35.731268 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:35.731179 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:35.731542 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:35.731519 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:41:36.884179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:36.884140 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:37.333973 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.333937 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq"] Nov 02 06:41:37.338381 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.338354 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.396099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.393589 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e5502564-openshift-default-dockercfg-6pljf\"" Nov 02 06:41:37.438184 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.427010 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-podinfo\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.438495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.438462 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-socket\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.446103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.438795 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-credential-socket\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.446342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.446317 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-data\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.446483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.446467 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-envoy\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.446641 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.446625 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istiod-ca-cert\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.446766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.446749 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9zhl\" (UniqueName: \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-kube-api-access-t9zhl\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.453429 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.451352 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-certs\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.453429 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.451436 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-token\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.584249 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.562988 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9zhl\" (UniqueName: \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-kube-api-access-t9zhl\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.693314 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.693152 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-certs\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.694164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.694143 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-token\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.694394 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.694378 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-podinfo\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.694602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.694586 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-socket\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.694889 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.694874 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-credential-socket\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.695004 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.694989 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-data\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.696804 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.696783 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-envoy\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.697044 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.697012 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istiod-ca-cert\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.697307 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.696148 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-socket\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.700307 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.694021 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-certs\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.700595 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.699621 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-credential-socket\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.704713 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.699825 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-data\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.704844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.679687 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:37.710996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.710973 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istiod-ca-cert\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.739414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.739259 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9zhl\" (UniqueName: \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-kube-api-access-t9zhl\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.739414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.739332 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq"] Nov 02 06:41:37.774792 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.774759 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-podinfo\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.777918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.777866 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-envoy\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.787126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.785990 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-token\") pod \"e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:37.911602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.911474 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:37.958420 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:37.958123 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:38.129116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:38.125795 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:39.047101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:39.043925 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq"] Nov 02 06:41:39.124128 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:39.124063 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:39.124371 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:39.124344 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:39.124456 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:39.124402 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:39.825375 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:39.825339 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" event={"ID":"a3992d0d-ee6b-48dc-a41a-dc2302cc0977","Type":"ContainerStarted","Data":"36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f"} Nov 02 06:41:39.825548 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:39.825410 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" event={"ID":"a3992d0d-ee6b-48dc-a41a-dc2302cc0977","Type":"ContainerStarted","Data":"08b2319344edf77b9c5d725e987dae0cbdce2e4ca9b90f7f46a49f3530f3cdc6"} Nov 02 06:41:39.959957 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:39.959924 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:39.973711 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:39.973658 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" podStartSLOduration=2.973645368 podStartE2EDuration="2.973645368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:41:39.969942706 +0000 UTC m=+1476.418319461" watchObservedRunningTime="2026-11-02 06:41:39.973645368 +0000 UTC m=+1476.422022123" Nov 02 06:41:40.328708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:40.328670 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:40.822524 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:40.822491 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:41.016102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.010679 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:41.068985 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.043710 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:41.357102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.353236 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:41.387177 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.387129 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8"] Nov 02 06:41:41.555166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.539363 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 06:41:41.673157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.640331 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-3e96be\"" Nov 02 06:41:41.673157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.640697 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-3e96be-dockercfg-pgzb7\"" Nov 02 06:41:41.673157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.640738 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8"] Nov 02 06:41:41.738565 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.738533 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp"] Nov 02 06:41:41.753184 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.753155 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 06:41:41.772340 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.772305 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp"] Nov 02 06:41:41.776069 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.775998 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff748fe-8003-46c3-a21a-e5c7f690e47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8\" (UID: \"eff748fe-8003-46c3-a21a-e5c7f690e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 06:41:41.776635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.776609 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56qq8\" (UniqueName: \"kubernetes.io/projected/eff748fe-8003-46c3-a21a-e5c7f690e47a-kube-api-access-56qq8\") pod \"payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8\" (UID: \"eff748fe-8003-46c3-a21a-e5c7f690e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 06:41:41.838297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.838234 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:41.840425 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.840334 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:41:41.878654 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.878622 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff748fe-8003-46c3-a21a-e5c7f690e47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8\" (UID: \"eff748fe-8003-46c3-a21a-e5c7f690e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 06:41:41.878827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.878670 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28pck\" (UniqueName: \"kubernetes.io/projected/b6d476df-b880-4cde-93db-1ce99a8ca8f0-kube-api-access-28pck\") pod \"payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp\" (UID: \"b6d476df-b880-4cde-93db-1ce99a8ca8f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 06:41:41.878827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.878742 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56qq8\" (UniqueName: \"kubernetes.io/projected/eff748fe-8003-46c3-a21a-e5c7f690e47a-kube-api-access-56qq8\") pod \"payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8\" (UID: \"eff748fe-8003-46c3-a21a-e5c7f690e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 06:41:41.878972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.878950 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d476df-b880-4cde-93db-1ce99a8ca8f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp\" (UID: \"b6d476df-b880-4cde-93db-1ce99a8ca8f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 06:41:41.879417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.879393 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff748fe-8003-46c3-a21a-e5c7f690e47a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8\" (UID: \"eff748fe-8003-46c3-a21a-e5c7f690e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 06:41:41.933697 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.933622 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56qq8\" (UniqueName: \"kubernetes.io/projected/eff748fe-8003-46c3-a21a-e5c7f690e47a-kube-api-access-56qq8\") pod \"payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8\" (UID: \"eff748fe-8003-46c3-a21a-e5c7f690e47a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 06:41:41.980262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.980224 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d476df-b880-4cde-93db-1ce99a8ca8f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp\" (UID: \"b6d476df-b880-4cde-93db-1ce99a8ca8f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 06:41:41.980453 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.980324 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28pck\" (UniqueName: \"kubernetes.io/projected/b6d476df-b880-4cde-93db-1ce99a8ca8f0-kube-api-access-28pck\") pod \"payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp\" (UID: \"b6d476df-b880-4cde-93db-1ce99a8ca8f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 06:41:41.981118 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:41.981092 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d476df-b880-4cde-93db-1ce99a8ca8f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp\" (UID: \"b6d476df-b880-4cde-93db-1ce99a8ca8f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 06:41:42.039708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.039627 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28pck\" (UniqueName: \"kubernetes.io/projected/b6d476df-b880-4cde-93db-1ce99a8ca8f0-kube-api-access-28pck\") pod \"payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp\" (UID: \"b6d476df-b880-4cde-93db-1ce99a8ca8f0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 06:41:42.078780 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.078748 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 06:41:42.114102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.113455 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 06:41:42.524333 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:41:42.524306 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeff748fe_8003_46c3_a21a_e5c7f690e47a.slice/crio-cd2529740c35005c763c450201dd4207f6f0f477e8becd51b6352020a2917fbb WatchSource:0}: Error finding container cd2529740c35005c763c450201dd4207f6f0f477e8becd51b6352020a2917fbb: Status 404 returned error can't find the container with id cd2529740c35005c763c450201dd4207f6f0f477e8becd51b6352020a2917fbb Nov 02 06:41:42.526294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.526269 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8"] Nov 02 06:41:42.571897 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:41:42.571873 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6d476df_b880_4cde_93db_1ce99a8ca8f0.slice/crio-9cebc4456ce61445f6f93f4b7a9f0dc0f92803cf9c028d1bcbadfffc3a2dc017 WatchSource:0}: Error finding container 9cebc4456ce61445f6f93f4b7a9f0dc0f92803cf9c028d1bcbadfffc3a2dc017: Status 404 returned error can't find the container with id 9cebc4456ce61445f6f93f4b7a9f0dc0f92803cf9c028d1bcbadfffc3a2dc017 Nov 02 06:41:42.573011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.572949 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp"] Nov 02 06:41:42.846182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.846066 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" event={"ID":"eff748fe-8003-46c3-a21a-e5c7f690e47a","Type":"ContainerStarted","Data":"03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820"} Nov 02 06:41:42.846182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.846143 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" event={"ID":"eff748fe-8003-46c3-a21a-e5c7f690e47a","Type":"ContainerStarted","Data":"cd2529740c35005c763c450201dd4207f6f0f477e8becd51b6352020a2917fbb"} Nov 02 06:41:42.846182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.846177 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 06:41:42.848032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.848001 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" event={"ID":"b6d476df-b880-4cde-93db-1ce99a8ca8f0","Type":"ContainerStarted","Data":"7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3"} Nov 02 06:41:42.848199 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.848058 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 06:41:42.848199 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.848109 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" event={"ID":"b6d476df-b880-4cde-93db-1ce99a8ca8f0","Type":"ContainerStarted","Data":"9cebc4456ce61445f6f93f4b7a9f0dc0f92803cf9c028d1bcbadfffc3a2dc017"} Nov 02 06:41:42.948670 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.948643 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:42.971131 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:42.971016 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" podStartSLOduration=1.970996366 podStartE2EDuration="1.970996366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:41:42.962301057 +0000 UTC m=+1479.410677821" watchObservedRunningTime="2026-11-02 06:41:42.970996366 +0000 UTC m=+1479.419373128" Nov 02 06:41:43.073468 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:43.073417 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" podStartSLOduration=2.073402906 podStartE2EDuration="2.073402906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:41:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:41:43.072428004 +0000 UTC m=+1479.520804785" watchObservedRunningTime="2026-11-02 06:41:43.073402906 +0000 UTC m=+1479.521779661" Nov 02 06:41:43.140944 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:43.140857 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:45.941098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:45.938125 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:46.558894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:46.558841 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:47.645782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:47.645748 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:48.044774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.044715 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:48.177513 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.177464 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t"] Nov 02 06:41:48.202947 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.202917 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t"] Nov 02 06:41:48.203286 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.203268 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:41:48.231930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.231839 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p5d6\" (UniqueName: \"kubernetes.io/projected/8b284776-73cc-4707-a5ea-b074892fc599-kube-api-access-9p5d6\") pod \"payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t\" (UID: \"8b284776-73cc-4707-a5ea-b074892fc599\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:41:48.232446 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.232406 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b284776-73cc-4707-a5ea-b074892fc599-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t\" (UID: \"8b284776-73cc-4707-a5ea-b074892fc599\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:41:48.237251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.237221 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7"] Nov 02 06:41:48.301390 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.301310 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-e5502564-dockercfg-wsjsn\"" Nov 02 06:41:48.301629 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.301612 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-e5502564\"" Nov 02 06:41:48.360327 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.360283 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b284776-73cc-4707-a5ea-b074892fc599-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t\" (UID: \"8b284776-73cc-4707-a5ea-b074892fc599\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:41:48.361013 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.360970 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p5d6\" (UniqueName: \"kubernetes.io/projected/8b284776-73cc-4707-a5ea-b074892fc599-kube-api-access-9p5d6\") pod \"payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t\" (UID: \"8b284776-73cc-4707-a5ea-b074892fc599\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:41:48.364724 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.364691 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b284776-73cc-4707-a5ea-b074892fc599-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t\" (UID: \"8b284776-73cc-4707-a5ea-b074892fc599\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:41:48.422364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.422330 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7"] Nov 02 06:41:48.422784 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.422764 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:41:48.447840 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.447773 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p5d6\" (UniqueName: \"kubernetes.io/projected/8b284776-73cc-4707-a5ea-b074892fc599-kube-api-access-9p5d6\") pod \"payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t\" (UID: \"8b284776-73cc-4707-a5ea-b074892fc599\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:41:48.585835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.572429 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:41:48.585835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.581846 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9t4g\" (UniqueName: \"kubernetes.io/projected/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-kube-api-access-m9t4g\") pod \"payload-processing-e2e-mt-e5502564-67bd495475-xtbx7\" (UID: \"02bbf1c2-fc80-4ec0-98b5-00258244e5ef\") " pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:41:48.585835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.581977 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-plugins-config-volume\") pod \"payload-processing-e2e-mt-e5502564-67bd495475-xtbx7\" (UID: \"02bbf1c2-fc80-4ec0-98b5-00258244e5ef\") " pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:41:48.683956 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.683898 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9t4g\" (UniqueName: \"kubernetes.io/projected/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-kube-api-access-m9t4g\") pod \"payload-processing-e2e-mt-e5502564-67bd495475-xtbx7\" (UID: \"02bbf1c2-fc80-4ec0-98b5-00258244e5ef\") " pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:41:48.685271 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.684057 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-plugins-config-volume\") pod \"payload-processing-e2e-mt-e5502564-67bd495475-xtbx7\" (UID: \"02bbf1c2-fc80-4ec0-98b5-00258244e5ef\") " pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:41:48.685340 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.685281 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-plugins-config-volume\") pod \"payload-processing-e2e-mt-e5502564-67bd495475-xtbx7\" (UID: \"02bbf1c2-fc80-4ec0-98b5-00258244e5ef\") " pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:41:48.748851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.748819 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9t4g\" (UniqueName: \"kubernetes.io/projected/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-kube-api-access-m9t4g\") pod \"payload-processing-e2e-mt-e5502564-67bd495475-xtbx7\" (UID: \"02bbf1c2-fc80-4ec0-98b5-00258244e5ef\") " pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:41:48.992249 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.992204 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:41:48.992425 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:48.992281 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:41:49.046437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:49.046400 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:41:49.074185 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:49.074155 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t"] Nov 02 06:41:49.074313 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:41:49.074222 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b284776_73cc_4707_a5ea_b074892fc599.slice/crio-c1652a926962bc47085e444dfe7e71ba092cd995aa2540bee7e258814984d641 WatchSource:0}: Error finding container c1652a926962bc47085e444dfe7e71ba092cd995aa2540bee7e258814984d641: Status 404 returned error can't find the container with id c1652a926962bc47085e444dfe7e71ba092cd995aa2540bee7e258814984d641 Nov 02 06:41:49.208040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:49.208004 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" event={"ID":"8b284776-73cc-4707-a5ea-b074892fc599","Type":"ContainerStarted","Data":"8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc"} Nov 02 06:41:49.208207 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:49.208069 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" event={"ID":"8b284776-73cc-4707-a5ea-b074892fc599","Type":"ContainerStarted","Data":"c1652a926962bc47085e444dfe7e71ba092cd995aa2540bee7e258814984d641"} Nov 02 06:41:49.208207 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:49.208131 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:41:49.320849 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:49.320740 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" podStartSLOduration=2.320721038 podStartE2EDuration="2.320721038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:41:49.316637109 +0000 UTC m=+1485.765013885" watchObservedRunningTime="2026-11-02 06:41:49.320721038 +0000 UTC m=+1485.769097795" Nov 02 06:41:49.424935 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:41:49.424898 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02bbf1c2_fc80_4ec0_98b5_00258244e5ef.slice/crio-d30eb0b04ff2034a6b208f5655b7dd7c1fdd5af6556224c769098652eeaac63f WatchSource:0}: Error finding container d30eb0b04ff2034a6b208f5655b7dd7c1fdd5af6556224c769098652eeaac63f: Status 404 returned error can't find the container with id d30eb0b04ff2034a6b208f5655b7dd7c1fdd5af6556224c769098652eeaac63f Nov 02 06:41:49.469729 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:49.430614 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7"] Nov 02 06:41:50.216890 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:50.216850 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" event={"ID":"02bbf1c2-fc80-4ec0-98b5-00258244e5ef","Type":"ContainerStarted","Data":"fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b"} Nov 02 06:41:50.217344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:50.216902 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" event={"ID":"02bbf1c2-fc80-4ec0-98b5-00258244e5ef","Type":"ContainerStarted","Data":"d30eb0b04ff2034a6b208f5655b7dd7c1fdd5af6556224c769098652eeaac63f"} Nov 02 06:41:50.217344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:50.217247 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:41:50.375579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:50.375522 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" podStartSLOduration=2.375507313 podStartE2EDuration="2.375507313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:41:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:41:50.327207207 +0000 UTC m=+1486.775583976" watchObservedRunningTime="2026-11-02 06:41:50.375507313 +0000 UTC m=+1486.823884068" Nov 02 06:41:50.711832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:50.711798 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:51.225574 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:51.225540 5025 generic.go:356] "Generic (PLEG): container finished" podID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerID="68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65" exitCode=0 Nov 02 06:41:51.226026 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:51.225608 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" event={"ID":"1f1a9eb4-dde1-488d-96ae-8b699703edab","Type":"ContainerDied","Data":"68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65"} Nov 02 06:41:51.226123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:51.226108 5025 scope.go:122] "RemoveContainer" containerID="68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65" Nov 02 06:41:51.227668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:51.227634 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:51.227787 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:51.227727 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:51.227852 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:51.227782 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:41:52.234309 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:52.234274 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" event={"ID":"1f1a9eb4-dde1-488d-96ae-8b699703edab","Type":"ContainerStarted","Data":"8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c"} Nov 02 06:41:52.831884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:52.831856 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:41:52.832056 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:52.832037 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 06:41:52.832116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:52.832093 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 06:41:53.831831 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:53.831797 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 06:41:53.832262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:53.831851 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 06:41:53.857691 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:53.857664 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 06:41:53.857867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:53.857731 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 06:41:54.832508 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:54.832475 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 06:41:54.832927 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:54.832544 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 06:41:55.283107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:55.281567 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:55.832470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:55.832437 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 06:41:55.832663 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:55.832502 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 06:41:56.831930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:56.831889 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body= Nov 02 06:41:56.832154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:56.831951 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" Nov 02 06:41:57.835800 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:57.835773 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:41:57.836288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:57.836268 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:41:57.836998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:57.836975 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:41:58.735021 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:58.734985 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:58.922946 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:58.922911 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:41:59.889901 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:41:59.889865 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:00.219878 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:00.219850 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:42:01.229459 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:01.229427 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:42:02.709273 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:42:02.709232 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:42:06.531228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:06.531191 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:09.035561 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:09.035491 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:14.182627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:14.182563 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:16.886094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:16.886031 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:17.282159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:17.282105 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:18.024857 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:18.024816 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:18.308827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:18.308742 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:19.310599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:19.310559 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:22.021138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:22.021068 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:24.013575 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:24.013543 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:24.322210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:24.312464 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:26.446637 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:26.446606 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:26.802445 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:26.802354 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:29.250052 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:29.249950 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:32.708756 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:42:32.708718 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:42:34.192123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:34.181206 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:34.396797 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:34.396758 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:34.578464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:34.578379 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:38.558168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:38.558120 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:39.534101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:39.526730 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:42.038115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.038085 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t"] Nov 02 06:42:42.041930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.041913 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.121215 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.121184 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t"] Nov 02 06:42:42.138628 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.138598 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3bf02ebf\"/\"openshift-service-ca.crt\"" Nov 02 06:42:42.138628 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.138629 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-3bf02ebf\"/\"kube-root-ca.crt\"" Nov 02 06:42:42.139633 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.139594 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3bf02ebf\"/\"e2e-lifecycle-model-3bf02ebf-kserve-self-signed-certs\"" Nov 02 06:42:42.139767 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.139642 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-3bf02ebf\"/\"default-dockercfg-ns4xq\"" Nov 02 06:42:42.165406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.165380 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tls-certs\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.165564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.165543 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tmp-dir\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.165755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.165729 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-model-cache\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.165811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.165791 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s9q6\" (UniqueName: \"kubernetes.io/projected/4878a04e-ddb9-4772-a068-3cdba2e83d5c-kube-api-access-5s9q6\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.165942 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.165887 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-home\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.165942 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.165925 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-dshm\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.225894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.203218 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:42.269335 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.269288 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tmp-dir\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.269536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.269480 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-model-cache\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.269611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.269544 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s9q6\" (UniqueName: \"kubernetes.io/projected/4878a04e-ddb9-4772-a068-3cdba2e83d5c-kube-api-access-5s9q6\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.269666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.269606 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-home\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.269666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.269639 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-dshm\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.269817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.269769 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tls-certs\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.295141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.293174 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tmp-dir\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.295141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.293489 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-model-cache\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.295141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.293730 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-home\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.309095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.306387 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tls-certs\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.325195 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.325147 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-dshm\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.347774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.347739 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s9q6\" (UniqueName: \"kubernetes.io/projected/4878a04e-ddb9-4772-a068-3cdba2e83d5c-kube-api-access-5s9q6\") pod \"e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.353586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.353530 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:42.386945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.386913 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb"] Nov 02 06:42:42.488671 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.488629 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h"] Nov 02 06:42:42.807096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.800962 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:42.922293 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:42.922156 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t"] Nov 02 06:42:43.158875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:43.134279 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:43.242137 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:43.242049 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" event={"ID":"4878a04e-ddb9-4772-a068-3cdba2e83d5c","Type":"ContainerStarted","Data":"9e74d6d73dce8c69c10fbc564411cb7c2494a0ddc581c41eda45e6b2335baf4c"} Nov 02 06:42:44.279624 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:44.279399 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" event={"ID":"4878a04e-ddb9-4772-a068-3cdba2e83d5c","Type":"ContainerStarted","Data":"e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73"} Nov 02 06:42:44.354206 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:44.354170 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:44.403129 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:44.402396 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" podStartSLOduration=3.402378512 podStartE2EDuration="3.402378512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:42:44.401963562 +0000 UTC m=+1540.850340331" watchObservedRunningTime="2026-11-02 06:42:44.402378512 +0000 UTC m=+1540.850755268" Nov 02 06:42:44.661243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:44.661092 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:44.854426 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:44.854396 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:45.040127 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:45.037207 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:47.387333 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:47.387264 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" podUID="04fc5642-8c5b-439b-ad74-a970672a1acc" containerName="payload-processing" containerID="cri-o://8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08" gracePeriod=25 Nov 02 06:42:47.499267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:47.499188 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" podUID="88d743ff-c4ed-4e97-9236-e337fdc88f06" containerName="payload-pre-processing" containerID="cri-o://eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993" gracePeriod=25 Nov 02 06:42:47.901495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:47.901167 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:42:47.940630 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:47.940578 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:42:48.043097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.038393 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d743ff-c4ed-4e97-9236-e337fdc88f06-kube-api-access-wx88c\" (UniqueName: \"kubernetes.io/projected/88d743ff-c4ed-4e97-9236-e337fdc88f06-kube-api-access-wx88c\") pod \"88d743ff-c4ed-4e97-9236-e337fdc88f06\" (UID: \"88d743ff-c4ed-4e97-9236-e337fdc88f06\") " Nov 02 06:42:48.043097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.038543 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04fc5642-8c5b-439b-ad74-a970672a1acc-kube-api-access-8cphm\" (UniqueName: \"kubernetes.io/projected/04fc5642-8c5b-439b-ad74-a970672a1acc-kube-api-access-8cphm\") pod \"04fc5642-8c5b-439b-ad74-a970672a1acc\" (UID: \"04fc5642-8c5b-439b-ad74-a970672a1acc\") " Nov 02 06:42:48.043097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.038710 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d743ff-c4ed-4e97-9236-e337fdc88f06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d743ff-c4ed-4e97-9236-e337fdc88f06-plugins-config-volume\") pod \"88d743ff-c4ed-4e97-9236-e337fdc88f06\" (UID: \"88d743ff-c4ed-4e97-9236-e337fdc88f06\") " Nov 02 06:42:48.043097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.038817 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04fc5642-8c5b-439b-ad74-a970672a1acc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fc5642-8c5b-439b-ad74-a970672a1acc-plugins-config-volume\") pod \"04fc5642-8c5b-439b-ad74-a970672a1acc\" (UID: \"04fc5642-8c5b-439b-ad74-a970672a1acc\") " Nov 02 06:42:48.064100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.055783 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04fc5642-8c5b-439b-ad74-a970672a1acc-plugins-config-volume" pod "04fc5642-8c5b-439b-ad74-a970672a1acc" (UID: "04fc5642-8c5b-439b-ad74-a970672a1acc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:42:48.064100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.056245 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d743ff-c4ed-4e97-9236-e337fdc88f06-plugins-config-volume" pod "88d743ff-c4ed-4e97-9236-e337fdc88f06" (UID: "88d743ff-c4ed-4e97-9236-e337fdc88f06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:42:48.080552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.080501 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d743ff-c4ed-4e97-9236-e337fdc88f06-kube-api-access-wx88c" pod "88d743ff-c4ed-4e97-9236-e337fdc88f06" (UID: "88d743ff-c4ed-4e97-9236-e337fdc88f06"). InnerVolumeSpecName "kube-api-access-wx88c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:42:48.080779 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.080757 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04fc5642-8c5b-439b-ad74-a970672a1acc-kube-api-access-8cphm" pod "04fc5642-8c5b-439b-ad74-a970672a1acc" (UID: "04fc5642-8c5b-439b-ad74-a970672a1acc"). InnerVolumeSpecName "kube-api-access-8cphm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:42:48.140952 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.140919 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx88c\" (UniqueName: \"kubernetes.io/projected/88d743ff-c4ed-4e97-9236-e337fdc88f06-kube-api-access-wx88c\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:42:48.140952 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.140947 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cphm\" (UniqueName: \"kubernetes.io/projected/04fc5642-8c5b-439b-ad74-a970672a1acc-kube-api-access-8cphm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:42:48.140952 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.140958 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88d743ff-c4ed-4e97-9236-e337fdc88f06-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:42:48.141248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.140967 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04fc5642-8c5b-439b-ad74-a970672a1acc-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:42:48.310036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.309945 5025 generic.go:356] "Generic (PLEG): container finished" podID="88d743ff-c4ed-4e97-9236-e337fdc88f06" containerID="eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993" exitCode=0 Nov 02 06:42:48.310036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.309987 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" Nov 02 06:42:48.310299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.310024 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" event={"ID":"88d743ff-c4ed-4e97-9236-e337fdc88f06","Type":"ContainerDied","Data":"eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993"} Nov 02 06:42:48.310299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.310092 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h" event={"ID":"88d743ff-c4ed-4e97-9236-e337fdc88f06","Type":"ContainerDied","Data":"a25d67d0df4ebec2bd3b332fb2b1ff78155b380f6fb620a37da62cbd644e11b6"} Nov 02 06:42:48.310299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.310127 5025 scope.go:122] "RemoveContainer" containerID="eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993" Nov 02 06:42:48.311562 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.311541 5025 generic.go:356] "Generic (PLEG): container finished" podID="04fc5642-8c5b-439b-ad74-a970672a1acc" containerID="8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08" exitCode=0 Nov 02 06:42:48.311672 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.311595 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" Nov 02 06:42:48.311672 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.311623 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" event={"ID":"04fc5642-8c5b-439b-ad74-a970672a1acc","Type":"ContainerDied","Data":"8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08"} Nov 02 06:42:48.311672 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.311670 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb" event={"ID":"04fc5642-8c5b-439b-ad74-a970672a1acc","Type":"ContainerDied","Data":"7c84b85b18d261a52e4794c430e7ab5cefd08f9ca1cabc0df5f470b66fdf976f"} Nov 02 06:42:48.320607 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.320590 5025 scope.go:122] "RemoveContainer" containerID="eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993" Nov 02 06:42:48.320910 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:42:48.320891 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993\": container with ID starting with eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993 not found: ID does not exist" containerID="eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993" Nov 02 06:42:48.320967 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.320917 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993"} err="failed to get container status \"eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993\": rpc error: code = NotFound desc = could not find container \"eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993\": container with ID starting with eb7da6ebdfa8dba0ea4d04cab1436a5663c5acfddf1b43fdad15bf697c165993 not found: ID does not exist" Nov 02 06:42:48.320967 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.320932 5025 scope.go:122] "RemoveContainer" containerID="8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08" Nov 02 06:42:48.331896 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.331879 5025 scope.go:122] "RemoveContainer" containerID="8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08" Nov 02 06:42:48.332188 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:42:48.332170 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08\": container with ID starting with 8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08 not found: ID does not exist" containerID="8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08" Nov 02 06:42:48.332246 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.332195 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08"} err="failed to get container status \"8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08\": rpc error: code = NotFound desc = could not find container \"8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08\": container with ID starting with 8065b880d802fc161bc66a59110b7c4fbdf2ad3efba5190aa7dbc9a0197f6c08 not found: ID does not exist" Nov 02 06:42:48.468857 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.468825 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb"] Nov 02 06:42:48.472550 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.472524 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fc9fdd1c-5864b69495-9lsjb"] Nov 02 06:42:48.577395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.577307 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h"] Nov 02 06:42:48.583453 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:48.583425 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fc9fdd1c-99ff8c475-c248h"] Nov 02 06:42:49.670249 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:49.670219 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04fc5642-8c5b-439b-ad74-a970672a1acc" path="/var/lib/kubelet/pods/04fc5642-8c5b-439b-ad74-a970672a1acc/volumes" Nov 02 06:42:49.670894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:49.670874 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d743ff-c4ed-4e97-9236-e337fdc88f06" path="/var/lib/kubelet/pods/88d743ff-c4ed-4e97-9236-e337fdc88f06/volumes" Nov 02 06:42:49.768102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:49.761473 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:52.354516 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:52.354484 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:52.368851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:52.368828 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:53.360572 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:53.360542 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:42:54.509663 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:54.509593 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:42:59.093103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:42:59.090087 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:00.015440 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.015408 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:00.461524 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.461485 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9"] Nov 02 06:43:00.462555 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.462535 5025 cpu_manager.go:414] "containerMap: removing container" podUID="88d743ff-c4ed-4e97-9236-e337fdc88f06" containerName="payload-pre-processing" Nov 02 06:43:00.462609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.462556 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d743ff-c4ed-4e97-9236-e337fdc88f06" containerName="payload-pre-processing" Nov 02 06:43:00.462650 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.462610 5025 cpu_manager.go:414] "containerMap: removing container" podUID="04fc5642-8c5b-439b-ad74-a970672a1acc" containerName="payload-processing" Nov 02 06:43:00.462650 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.462618 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04fc5642-8c5b-439b-ad74-a970672a1acc" containerName="payload-processing" Nov 02 06:43:00.462779 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.462767 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d743ff-c4ed-4e97-9236-e337fdc88f06" containerName="payload-pre-processing" Nov 02 06:43:00.462821 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.462792 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="04fc5642-8c5b-439b-ad74-a970672a1acc" containerName="payload-processing" Nov 02 06:43:00.467161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.467139 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.521788 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.521753 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8ecb23-openshift-default-dockercfg-7tcjt\"" Nov 02 06:43:00.564190 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.564147 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-credential-socket\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.564392 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.564215 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-envoy\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.564392 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.564290 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-token\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.564392 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.564370 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-certs\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.564653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.564396 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfcmn\" (UniqueName: \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-kube-api-access-rfcmn\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.564653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.564552 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-data\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.564653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.564644 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istiod-ca-cert\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.564817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.564748 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-socket\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.564817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.564788 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-podinfo\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.574829 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.574790 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9"] Nov 02 06:43:00.623737 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.623701 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:00.709712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.699296 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-data\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.710138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.710101 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istiod-ca-cert\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.711329 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.711302 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-socket\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.711659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.711592 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-podinfo\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.711908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.711867 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-credential-socket\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.712158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.712067 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-envoy\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.712368 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.712351 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-token\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.712545 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.712503 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-certs\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.712658 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.712643 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfcmn\" (UniqueName: \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-kube-api-access-rfcmn\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.713408 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.711133 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istiod-ca-cert\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.714057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.714014 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-socket\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.714233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.707938 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-data\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.716723 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.716670 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-credential-socket\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.717124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.717104 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-certs\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.749320 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.747971 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-podinfo\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.749823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.749801 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-envoy\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.818101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.809725 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-token\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.818343 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.813794 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfcmn\" (UniqueName: \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-kube-api-access-rfcmn\") pod \"e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:00.923097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:00.923045 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:01.083301 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:01.082792 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:01.556203 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:01.556172 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9"] Nov 02 06:43:01.556828 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:43:01.556796 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e296f80_34b3_44ad_b978_bcd9b99f19c1.slice/crio-2c3bdad613707541a23f50cf6c1a4470d24bcaad2ce01277622f3130bdd35b98 WatchSource:0}: Error finding container 2c3bdad613707541a23f50cf6c1a4470d24bcaad2ce01277622f3130bdd35b98: Status 404 returned error can't find the container with id 2c3bdad613707541a23f50cf6c1a4470d24bcaad2ce01277622f3130bdd35b98 Nov 02 06:43:01.559680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:01.559645 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:01.559781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:01.559712 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:01.559781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:01.559740 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:02.437403 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:02.437356 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" event={"ID":"8e296f80-34b3-44ad-b978-bcd9b99f19c1","Type":"ContainerStarted","Data":"efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244"} Nov 02 06:43:02.437618 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:02.437420 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" event={"ID":"8e296f80-34b3-44ad-b978-bcd9b99f19c1","Type":"ContainerStarted","Data":"2c3bdad613707541a23f50cf6c1a4470d24bcaad2ce01277622f3130bdd35b98"} Nov 02 06:43:02.599576 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:02.599510 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" podStartSLOduration=2.599491336 podStartE2EDuration="2.599491336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:43:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:43:02.596814082 +0000 UTC m=+1559.045190864" watchObservedRunningTime="2026-11-02 06:43:02.599491336 +0000 UTC m=+1559.047868093" Nov 02 06:43:02.784180 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:43:02.782282 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:43:03.083538 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:03.083439 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:03.090506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:03.090476 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:03.444833 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:03.444800 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:03.445977 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:03.445954 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:43:03.715160 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:03.714909 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:05.406423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:05.406387 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:05.617552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:05.617514 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:08.613868 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:08.613826 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:08.849132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:08.811652 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:09.081845 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.081798 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:09.186102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.134310 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg"] Nov 02 06:43:09.375139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.373313 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg"] Nov 02 06:43:09.375139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.373444 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:43:09.424483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.420115 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg\" (UID: \"6aee84db-e7cf-41f6-9e44-e981dacfb8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:43:09.424483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.420231 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhjkg\" (UniqueName: \"kubernetes.io/projected/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-kube-api-access-dhjkg\") pod \"payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg\" (UID: \"6aee84db-e7cf-41f6-9e44-e981dacfb8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:43:09.433157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.431782 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq"] Nov 02 06:43:09.439828 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.439269 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:43:09.470863 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.470821 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq"] Nov 02 06:43:09.480627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.480438 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8ecb23\"" Nov 02 06:43:09.480627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.480586 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8ecb23-dockercfg-h58v8\"" Nov 02 06:43:09.533126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.522014 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq\" (UID: \"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:43:09.533126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.522229 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm9kj\" (UniqueName: \"kubernetes.io/projected/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-kube-api-access-zm9kj\") pod \"payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq\" (UID: \"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:43:09.533126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.522288 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg\" (UID: \"6aee84db-e7cf-41f6-9e44-e981dacfb8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:43:09.533126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.522371 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhjkg\" (UniqueName: \"kubernetes.io/projected/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-kube-api-access-dhjkg\") pod \"payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg\" (UID: \"6aee84db-e7cf-41f6-9e44-e981dacfb8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:43:09.535954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.535884 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg\" (UID: \"6aee84db-e7cf-41f6-9e44-e981dacfb8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:43:09.685963 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.638052 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm9kj\" (UniqueName: \"kubernetes.io/projected/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-kube-api-access-zm9kj\") pod \"payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq\" (UID: \"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:43:09.687018 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.686985 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq\" (UID: \"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:43:09.687334 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.685865 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:43:09.688926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.647456 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhjkg\" (UniqueName: \"kubernetes.io/projected/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-kube-api-access-dhjkg\") pod \"payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg\" (UID: \"6aee84db-e7cf-41f6-9e44-e981dacfb8cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:43:09.689955 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.689928 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq\" (UID: \"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:43:09.757394 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.757358 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:43:09.771008 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.770941 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm9kj\" (UniqueName: \"kubernetes.io/projected/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-kube-api-access-zm9kj\") pod \"payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq\" (UID: \"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb\") " pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:43:09.813132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.795346 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:43:09.844976 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:09.844942 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:43:10.251480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:10.251448 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg"] Nov 02 06:43:10.252346 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:43:10.252314 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aee84db_e7cf_41f6_9e44_e981dacfb8cc.slice/crio-836f1ecd1b6e8f7a76ebfc4a56af0523fb55efbafbd6a7f75d350334107c464d WatchSource:0}: Error finding container 836f1ecd1b6e8f7a76ebfc4a56af0523fb55efbafbd6a7f75d350334107c464d: Status 404 returned error can't find the container with id 836f1ecd1b6e8f7a76ebfc4a56af0523fb55efbafbd6a7f75d350334107c464d Nov 02 06:43:10.315452 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:10.315423 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq"] Nov 02 06:43:10.316407 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:43:10.316381 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f13cb1_b17f_4d7a_ab25_bcc430cd0ceb.slice/crio-9ff3464c5ea4e3592e75d566a138dd407ffdcb392b4233b7f44ffe5f85105856 WatchSource:0}: Error finding container 9ff3464c5ea4e3592e75d566a138dd407ffdcb392b4233b7f44ffe5f85105856: Status 404 returned error can't find the container with id 9ff3464c5ea4e3592e75d566a138dd407ffdcb392b4233b7f44ffe5f85105856 Nov 02 06:43:10.730468 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:10.730411 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" event={"ID":"6aee84db-e7cf-41f6-9e44-e981dacfb8cc","Type":"ContainerStarted","Data":"58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57"} Nov 02 06:43:10.730952 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:10.730686 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:43:10.730952 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:10.730734 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" event={"ID":"6aee84db-e7cf-41f6-9e44-e981dacfb8cc","Type":"ContainerStarted","Data":"836f1ecd1b6e8f7a76ebfc4a56af0523fb55efbafbd6a7f75d350334107c464d"} Nov 02 06:43:10.732165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:10.732133 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" event={"ID":"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb","Type":"ContainerStarted","Data":"7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91"} Nov 02 06:43:10.732326 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:10.732186 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" event={"ID":"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb","Type":"ContainerStarted","Data":"9ff3464c5ea4e3592e75d566a138dd407ffdcb392b4233b7f44ffe5f85105856"} Nov 02 06:43:10.732326 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:10.732296 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:43:10.829033 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:10.828983 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:10.861915 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:10.861862 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" podStartSLOduration=2.86184899 podStartE2EDuration="2.86184899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:43:10.857496148 +0000 UTC m=+1567.305872916" watchObservedRunningTime="2026-11-02 06:43:10.86184899 +0000 UTC m=+1567.310225746" Nov 02 06:43:10.969876 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:10.969814 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" podStartSLOduration=1.969793313 podStartE2EDuration="1.969793313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:43:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:43:10.966957453 +0000 UTC m=+1567.415334218" watchObservedRunningTime="2026-11-02 06:43:10.969793313 +0000 UTC m=+1567.418170070" Nov 02 06:43:11.943010 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:11.942974 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:12.126963 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:12.126927 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:14.131748 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:14.131704 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:15.881584 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:15.881552 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:16.167835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:16.167803 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:18.194688 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.194654 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244"] Nov 02 06:43:18.195105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.194879 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" podUID="ee74d7a3-a68f-44a8-bb32-8bd61cdefe62" containerName="maas-api" containerID="cri-o://70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb" gracePeriod=30 Nov 02 06:43:18.354883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.354847 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:18.700812 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.700404 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:43:18.781220 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.777369 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-kube-api-access-rzcxw\" (UniqueName: \"kubernetes.io/projected/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-kube-api-access-rzcxw\") pod \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " Nov 02 06:43:18.781220 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.777495 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-openshift-service-ca\") pod \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " Nov 02 06:43:18.781220 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.777612 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-maas-api-tls\") pod \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " Nov 02 06:43:18.781220 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.777673 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-metrics-certs\") pod \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\" (UID: \"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62\") " Nov 02 06:43:18.781220 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.779574 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-openshift-service-ca" pod "ee74d7a3-a68f-44a8-bb32-8bd61cdefe62" (UID: "ee74d7a3-a68f-44a8-bb32-8bd61cdefe62"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:18.782790 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.782233 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:18.863103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.863037 5025 generic.go:356] "Generic (PLEG): container finished" podID="ee74d7a3-a68f-44a8-bb32-8bd61cdefe62" containerID="70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb" exitCode=0 Nov 02 06:43:18.869296 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.869253 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" event={"ID":"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62","Type":"ContainerDied","Data":"70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb"} Nov 02 06:43:18.869587 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.869546 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" event={"ID":"ee74d7a3-a68f-44a8-bb32-8bd61cdefe62","Type":"ContainerDied","Data":"9c5ba69bd3d390f6004ff4810d9566736852f0065330d7851aa6099136f65d83"} Nov 02 06:43:18.869751 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.869736 5025 scope.go:122] "RemoveContainer" containerID="70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb" Nov 02 06:43:18.870052 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.870039 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244" Nov 02 06:43:18.892133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.881995 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-maas-api-tls" pod "ee74d7a3-a68f-44a8-bb32-8bd61cdefe62" (UID: "ee74d7a3-a68f-44a8-bb32-8bd61cdefe62"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:43:18.892133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.882183 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-kube-api-access-rzcxw" pod "ee74d7a3-a68f-44a8-bb32-8bd61cdefe62" (UID: "ee74d7a3-a68f-44a8-bb32-8bd61cdefe62"). InnerVolumeSpecName "kube-api-access-rzcxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:18.892133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.882297 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-metrics-certs" pod "ee74d7a3-a68f-44a8-bb32-8bd61cdefe62" (UID: "ee74d7a3-a68f-44a8-bb32-8bd61cdefe62"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:43:18.892133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.884435 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzcxw\" (UniqueName: \"kubernetes.io/projected/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-kube-api-access-rzcxw\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:18.892133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.884467 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:18.892133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.884484 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:18.927070 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.927042 5025 scope.go:122] "RemoveContainer" containerID="70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb" Nov 02 06:43:18.927557 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:43:18.927525 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb\": container with ID starting with 70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb not found: ID does not exist" containerID="70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb" Nov 02 06:43:18.927634 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:18.927572 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb"} err="failed to get container status \"70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb\": rpc error: code = NotFound desc = could not find container \"70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb\": container with ID starting with 70a17adbdfbde7edd40fa6539bafae64e3d99d25375bb97bdfc77bfcd21a3feb not found: ID does not exist" Nov 02 06:43:19.017402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:19.017353 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:19.309354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:19.199041 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:19.477155 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:19.466343 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5"] Nov 02 06:43:19.477372 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:19.477180 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t"] Nov 02 06:43:19.477372 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:19.477221 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244"] Nov 02 06:43:19.477498 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:19.477397 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" containerID="cri-o://299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75" gracePeriod=30 Nov 02 06:43:19.486758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:19.477664 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerName="istio-proxy" containerID="cri-o://975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0" gracePeriod=30 Nov 02 06:43:19.625023 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:19.624988 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fc9fdd1c-7b7dc4b9cc-hx244"] Nov 02 06:43:20.170228 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.170041 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:20.186033 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.186003 5025 generic.go:356] "Generic (PLEG): container finished" podID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerID="5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218" exitCode=0 Nov 02 06:43:20.187501 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.187465 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee74d7a3-a68f-44a8-bb32-8bd61cdefe62" path="/var/lib/kubelet/pods/ee74d7a3-a68f-44a8-bb32-8bd61cdefe62/volumes" Nov 02 06:43:20.187891 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.187869 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" event={"ID":"9abe717d-5020-42af-9ad7-ab8c166f6936","Type":"ContainerDied","Data":"5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218"} Nov 02 06:43:20.188379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.188360 5025 scope.go:122] "RemoveContainer" containerID="5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218" Nov 02 06:43:20.190137 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.190111 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z"] Nov 02 06:43:20.190313 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.190284 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:20.190401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.190322 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" containerID="cri-o://4bdd630cfe5ca1d893f423e4be03fcddef22e90e69e62044638b0931a2e4d6a7" gracePeriod=30 Nov 02 06:43:20.190401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.190369 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:20.190525 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.190422 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:20.355885 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.355828 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:20.553751 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.553664 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:20.689015 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.688979 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:43:20.797334 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.797300 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:20.979785 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:20.979733 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:21.207160 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:21.206385 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:21.254148 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:21.238457 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" event={"ID":"9abe717d-5020-42af-9ad7-ab8c166f6936","Type":"ContainerStarted","Data":"fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff"} Nov 02 06:43:21.432098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:21.423568 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:43:21.432098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:21.427550 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 06:43:21.432098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:21.427614 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 06:43:21.758100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:21.756280 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:43:21.782918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:21.782885 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:43:22.357351 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.357317 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:22.416718 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.416688 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 06:43:22.416923 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.416748 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 06:43:22.550233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.550200 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:22.854521 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.854485 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66"] Nov 02 06:43:22.855277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.855258 5025 cpu_manager.go:414] "containerMap: removing container" podUID="ee74d7a3-a68f-44a8-bb32-8bd61cdefe62" containerName="maas-api" Nov 02 06:43:22.855277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.855278 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee74d7a3-a68f-44a8-bb32-8bd61cdefe62" containerName="maas-api" Nov 02 06:43:22.855420 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.855410 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee74d7a3-a68f-44a8-bb32-8bd61cdefe62" containerName="maas-api" Nov 02 06:43:22.859293 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.859270 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:22.910597 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.910566 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-70898f-openshift-default-dockercfg-xx946\"" Nov 02 06:43:22.972701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.970682 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qjbw\" (UniqueName: \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-kube-api-access-4qjbw\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:22.972701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.970797 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-certs\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:22.972701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.970898 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-data\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:22.972701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.970924 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istiod-ca-cert\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:22.972701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.971038 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-credential-socket\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:22.972701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.971212 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-socket\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:22.972701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.971524 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-envoy\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:22.972701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.971601 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-podinfo\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:22.972701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.971657 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-token\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:22.976001 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:22.975263 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66"] Nov 02 06:43:23.054126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.053178 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:23.110169 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.079379 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-podinfo\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.110169 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.079472 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-token\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.110169 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.079602 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qjbw\" (UniqueName: \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-kube-api-access-4qjbw\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.110169 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.079715 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-certs\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.110169 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.079818 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-data\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.110169 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.079849 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istiod-ca-cert\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.110169 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.079931 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-credential-socket\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.110169 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.079995 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-socket\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.110169 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.080044 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-envoy\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.162099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.141813 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-certs\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.187097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.185284 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istiod-ca-cert\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.187097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.185569 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-data\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.187097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.185806 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-credential-socket\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.187097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.186589 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-socket\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.275952 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.274832 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-envoy\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.364205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.361181 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-podinfo\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.402341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.376961 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-token\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.409102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.407033 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qjbw\" (UniqueName: \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-kube-api-access-4qjbw\") pod \"e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:23.445103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.420843 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 06:43:23.445103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.420904 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 06:43:23.445103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.443624 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:23.492112 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:23.485443 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:24.030205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:24.030173 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66"] Nov 02 06:43:24.030556 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:43:24.030509 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a4894f3_c78f_4acb_81b3_09dc3abcb0f7.slice/crio-eb5aee44d4dc3d761dcd75fa4ac2e2ea0e002a486c1c11e09fdd06b6fa7ba1cf WatchSource:0}: Error finding container eb5aee44d4dc3d761dcd75fa4ac2e2ea0e002a486c1c11e09fdd06b6fa7ba1cf: Status 404 returned error can't find the container with id eb5aee44d4dc3d761dcd75fa4ac2e2ea0e002a486c1c11e09fdd06b6fa7ba1cf Nov 02 06:43:24.033281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:24.033243 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:24.033384 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:24.033309 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:24.033384 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:24.033338 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:24.417722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:24.417687 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 06:43:24.417907 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:24.417754 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 06:43:24.453418 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:24.453382 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" event={"ID":"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7","Type":"ContainerStarted","Data":"3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879"} Nov 02 06:43:24.453592 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:24.453445 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" event={"ID":"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7","Type":"ContainerStarted","Data":"eb5aee44d4dc3d761dcd75fa4ac2e2ea0e002a486c1c11e09fdd06b6fa7ba1cf"} Nov 02 06:43:24.489037 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:24.488985 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:24.573684 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:24.573631 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" podStartSLOduration=2.573612873 podStartE2EDuration="2.573612873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:43:24.570455643 +0000 UTC m=+1581.018832401" watchObservedRunningTime="2026-11-02 06:43:24.573612873 +0000 UTC m=+1581.021989629" Nov 02 06:43:25.193829 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.182726 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:43:25.270172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.269699 5025 patch_prober.go:29] interesting pod/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 02 06:43:25.270172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.269756 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 02 06:43:25.324447 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.324417 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:43:25.342136 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.342067 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-kube-api-access-pjbkv\" (UniqueName: \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-kube-api-access-pjbkv\") pod \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " Nov 02 06:43:25.342279 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.342161 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-certs\") pod \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " Nov 02 06:43:25.342279 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.342241 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-podinfo\") pod \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " Nov 02 06:43:25.342279 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.342269 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-envoy\") pod \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " Nov 02 06:43:25.342450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.342339 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-credential-socket\") pod \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " Nov 02 06:43:25.342450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.342388 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-token\" (UniqueName: \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-token\") pod \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " Nov 02 06:43:25.342548 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.342462 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istiod-ca-cert\") pod \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " Nov 02 06:43:25.342548 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.342505 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-socket\") pod \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " Nov 02 06:43:25.342667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.342646 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-data\") pod \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\" (UID: \"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f\") " Nov 02 06:43:25.344035 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.344004 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-data" pod "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" (UID: "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.344304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.344279 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-credential-socket" pod "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" (UID: "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.348552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.348245 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-certs" pod "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" (UID: "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.355738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.349257 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:25.355738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.349063 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-socket" pod "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" (UID: "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.355738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.352176 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istiod-ca-cert" pod "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" (UID: "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:25.355738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.352192 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-token" pod "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" (UID: "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:25.355738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.354490 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-kube-api-access-pjbkv" pod "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" (UID: "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f"). InnerVolumeSpecName "kube-api-access-pjbkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:25.366405 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.366340 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-envoy" pod "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" (UID: "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.369725 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.369679 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-podinfo" pod "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" (UID: "6ffc5c9e-1a15-4d4c-9762-18fc84808b8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:25.420095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.419511 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 06:43:25.420095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.419573 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 06:43:25.444743 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.444704 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-socket\") pod \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " Nov 02 06:43:25.444907 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.444762 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-envoy\") pod \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " Nov 02 06:43:25.444907 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.444868 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-token\" (UniqueName: \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-token\") pod \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " Nov 02 06:43:25.444980 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.444910 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-kube-api-access-8ssb4\" (UniqueName: \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-kube-api-access-8ssb4\") pod \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " Nov 02 06:43:25.445042 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.445026 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-credential-socket\") pod \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " Nov 02 06:43:25.445219 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.445199 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istiod-ca-cert\") pod \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " Nov 02 06:43:25.445321 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.445307 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-certs\") pod \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " Nov 02 06:43:25.446233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.445378 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-podinfo\") pod \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " Nov 02 06:43:25.446233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.445448 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-data\") pod \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\" (UID: \"b575a0a5-5c30-4169-bc8b-e2aeb933d15f\") " Nov 02 06:43:25.447190 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447062 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjbkv\" (UniqueName: \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-kube-api-access-pjbkv\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.447190 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447120 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-socket" pod "b575a0a5-5c30-4169-bc8b-e2aeb933d15f" (UID: "b575a0a5-5c30-4169-bc8b-e2aeb933d15f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.447190 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447133 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.447402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447198 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.447402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447215 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.447402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447231 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.447402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447247 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.447402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447251 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-credential-socket" pod "b575a0a5-5c30-4169-bc8b-e2aeb933d15f" (UID: "b575a0a5-5c30-4169-bc8b-e2aeb933d15f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.447402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447261 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.447402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447277 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.447402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447292 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.447855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.447406 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istiod-ca-cert" pod "b575a0a5-5c30-4169-bc8b-e2aeb933d15f" (UID: "b575a0a5-5c30-4169-bc8b-e2aeb933d15f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:25.448217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.448142 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-data" pod "b575a0a5-5c30-4169-bc8b-e2aeb933d15f" (UID: "b575a0a5-5c30-4169-bc8b-e2aeb933d15f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.448360 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.448292 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-certs" pod "b575a0a5-5c30-4169-bc8b-e2aeb933d15f" (UID: "b575a0a5-5c30-4169-bc8b-e2aeb933d15f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.450342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.450314 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-token" pod "b575a0a5-5c30-4169-bc8b-e2aeb933d15f" (UID: "b575a0a5-5c30-4169-bc8b-e2aeb933d15f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:25.450728 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.450703 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-envoy" pod "b575a0a5-5c30-4169-bc8b-e2aeb933d15f" (UID: "b575a0a5-5c30-4169-bc8b-e2aeb933d15f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.451166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.451143 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-podinfo" pod "b575a0a5-5c30-4169-bc8b-e2aeb933d15f" (UID: "b575a0a5-5c30-4169-bc8b-e2aeb933d15f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:25.461678 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.460416 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-kube-api-access-8ssb4" pod "b575a0a5-5c30-4169-bc8b-e2aeb933d15f" (UID: "b575a0a5-5c30-4169-bc8b-e2aeb933d15f"). InnerVolumeSpecName "kube-api-access-8ssb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:25.473040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.473006 5025 generic.go:356] "Generic (PLEG): container finished" podID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerID="299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75" exitCode=0 Nov 02 06:43:25.473499 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.473260 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" event={"ID":"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f","Type":"ContainerDied","Data":"299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75"} Nov 02 06:43:25.473499 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.473333 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" event={"ID":"6ffc5c9e-1a15-4d4c-9762-18fc84808b8f","Type":"ContainerDied","Data":"b8ea68272b57f4a258e30ab66d0da9c45b1ba6fbb41bfc38dc3ebc032922273f"} Nov 02 06:43:25.473499 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.473366 5025 scope.go:122] "RemoveContainer" containerID="299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75" Nov 02 06:43:25.476167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.476134 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t" Nov 02 06:43:25.478817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.476366 5025 generic.go:356] "Generic (PLEG): container finished" podID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerID="975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0" exitCode=0 Nov 02 06:43:25.478817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.476584 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" event={"ID":"b575a0a5-5c30-4169-bc8b-e2aeb933d15f","Type":"ContainerDied","Data":"975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0"} Nov 02 06:43:25.478817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.476629 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" Nov 02 06:43:25.478817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.476628 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" event={"ID":"b575a0a5-5c30-4169-bc8b-e2aeb933d15f","Type":"ContainerDied","Data":"00d3b37eaa43f289f8c73af0887f9d6ae9ae75c422bccab32c9079a397af35e6"} Nov 02 06:43:25.478817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.478632 5025 generic.go:356] "Generic (PLEG): container finished" podID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerID="4bdd630cfe5ca1d893f423e4be03fcddef22e90e69e62044638b0931a2e4d6a7" exitCode=0 Nov 02 06:43:25.479791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.479492 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" event={"ID":"c41dade0-68ae-4d60-99de-33151bf4b2b8","Type":"ContainerDied","Data":"4bdd630cfe5ca1d893f423e4be03fcddef22e90e69e62044638b0931a2e4d6a7"} Nov 02 06:43:25.503116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.503017 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:43:25.516372 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.516303 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:25.531903 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.531814 5025 scope.go:122] "RemoveContainer" containerID="299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75" Nov 02 06:43:25.532207 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.532153 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:25.532312 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:43:25.532278 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75\": container with ID starting with 299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75 not found: ID does not exist" containerID="299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75" Nov 02 06:43:25.532378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.532312 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75"} err="failed to get container status \"299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75\": rpc error: code = NotFound desc = could not find container \"299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75\": container with ID starting with 299238258b6f7694338cef28d682ffe612cc9bc343ad4fdc892e2e6287d0bb75 not found: ID does not exist" Nov 02 06:43:25.532378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.532333 5025 scope.go:122] "RemoveContainer" containerID="975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0" Nov 02 06:43:25.548755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.548487 5025 scope.go:122] "RemoveContainer" containerID="975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0" Nov 02 06:43:25.548901 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.548877 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.548994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.548903 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.548994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.548920 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.548994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.548949 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ssb4\" (UniqueName: \"kubernetes.io/projected/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-kube-api-access-8ssb4\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.548994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.548971 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.548994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.548987 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.549200 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.549002 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.549200 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.549018 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.549200 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.549033 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b575a0a5-5c30-4169-bc8b-e2aeb933d15f-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.549200 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:43:25.549125 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0\": container with ID starting with 975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0 not found: ID does not exist" containerID="975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0" Nov 02 06:43:25.549200 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.549158 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0"} err="failed to get container status \"975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0\": rpc error: code = NotFound desc = could not find container \"975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0\": container with ID starting with 975252eba51cc9c1ec9a5db8673b5b5ce14aa03769abbd7e8e8d66e5b9edf7a0 not found: ID does not exist" Nov 02 06:43:25.661689 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.661599 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-kube-api-access-v9wzh\" (UniqueName: \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-kube-api-access-v9wzh\") pod \"c41dade0-68ae-4d60-99de-33151bf4b2b8\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " Nov 02 06:43:25.662116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.662090 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-socket\") pod \"c41dade0-68ae-4d60-99de-33151bf4b2b8\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " Nov 02 06:43:25.662450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.662423 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-envoy\") pod \"c41dade0-68ae-4d60-99de-33151bf4b2b8\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " Nov 02 06:43:25.662610 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.662593 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c41dade0-68ae-4d60-99de-33151bf4b2b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41dade0-68ae-4d60-99de-33151bf4b2b8-istiod-ca-cert\") pod \"c41dade0-68ae-4d60-99de-33151bf4b2b8\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " Nov 02 06:43:25.662718 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.662703 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-credential-socket\") pod \"c41dade0-68ae-4d60-99de-33151bf4b2b8\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " Nov 02 06:43:25.662900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.662886 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-podinfo\") pod \"c41dade0-68ae-4d60-99de-33151bf4b2b8\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " Nov 02 06:43:25.663009 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.662993 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-token\" (UniqueName: \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-token\") pod \"c41dade0-68ae-4d60-99de-33151bf4b2b8\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " Nov 02 06:43:25.663137 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.663120 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-data\") pod \"c41dade0-68ae-4d60-99de-33151bf4b2b8\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " Nov 02 06:43:25.663337 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.663315 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-certs\") pod \"c41dade0-68ae-4d60-99de-33151bf4b2b8\" (UID: \"c41dade0-68ae-4d60-99de-33151bf4b2b8\") " Nov 02 06:43:25.677613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.677551 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41dade0-68ae-4d60-99de-33151bf4b2b8-istiod-ca-cert" pod "c41dade0-68ae-4d60-99de-33151bf4b2b8" (UID: "c41dade0-68ae-4d60-99de-33151bf4b2b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:25.679356 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.678652 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-socket" pod "c41dade0-68ae-4d60-99de-33151bf4b2b8" (UID: "c41dade0-68ae-4d60-99de-33151bf4b2b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.695310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.681486 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-data" pod "c41dade0-68ae-4d60-99de-33151bf4b2b8" (UID: "c41dade0-68ae-4d60-99de-33151bf4b2b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.695310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.681673 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-certs" pod "c41dade0-68ae-4d60-99de-33151bf4b2b8" (UID: "c41dade0-68ae-4d60-99de-33151bf4b2b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.695310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.682108 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.695310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.682132 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c41dade0-68ae-4d60-99de-33151bf4b2b8-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.695310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.682145 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.695310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.682160 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.695310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.687342 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-credential-socket" pod "c41dade0-68ae-4d60-99de-33151bf4b2b8" (UID: "c41dade0-68ae-4d60-99de-33151bf4b2b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.727027 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.726972 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-envoy" pod "c41dade0-68ae-4d60-99de-33151bf4b2b8" (UID: "c41dade0-68ae-4d60-99de-33151bf4b2b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:25.852670 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.852629 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.852670 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.852673 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c41dade0-68ae-4d60-99de-33151bf4b2b8-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.858746 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.858712 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-token" pod "c41dade0-68ae-4d60-99de-33151bf4b2b8" (UID: "c41dade0-68ae-4d60-99de-33151bf4b2b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:25.858963 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.858712 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-podinfo" pod "c41dade0-68ae-4d60-99de-33151bf4b2b8" (UID: "c41dade0-68ae-4d60-99de-33151bf4b2b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:25.859037 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.858814 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-kube-api-access-v9wzh" pod "c41dade0-68ae-4d60-99de-33151bf4b2b8" (UID: "c41dade0-68ae-4d60-99de-33151bf4b2b8"). InnerVolumeSpecName "kube-api-access-v9wzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:25.954721 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.954684 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9wzh\" (UniqueName: \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-kube-api-access-v9wzh\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.954721 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.954716 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.954721 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.954727 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c41dade0-68ae-4d60-99de-33151bf4b2b8-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:25.989597 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.989561 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5"] Nov 02 06:43:25.999458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:25.999426 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5"] Nov 02 06:43:26.110932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.110895 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t"] Nov 02 06:43:26.116940 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.116910 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-dcs7t"] Nov 02 06:43:26.225951 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.225920 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:43:26.226352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.225986 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-8n8m5" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:43:26.417180 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.417147 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Nov 02 06:43:26.417352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.417198 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Nov 02 06:43:26.513644 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.513570 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" Nov 02 06:43:26.515760 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.515725 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z" event={"ID":"c41dade0-68ae-4d60-99de-33151bf4b2b8","Type":"ContainerDied","Data":"57bc1c0155aace7485bdacb05ca37e358d8983ced8026538e9568315556a360d"} Nov 02 06:43:26.516378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.516358 5025 scope.go:122] "RemoveContainer" containerID="4bdd630cfe5ca1d893f423e4be03fcddef22e90e69e62044638b0931a2e4d6a7" Nov 02 06:43:26.520429 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.520410 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:26.521925 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.521908 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:43:26.677761 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.677730 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z"] Nov 02 06:43:26.705754 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:26.705716 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-vgd6z"] Nov 02 06:43:27.316616 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:27.316576 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:27.426409 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:27.426373 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:43:27.426911 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:27.426880 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:43:27.427880 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:27.427845 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 06:43:27.513763 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:27.513733 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:27.674286 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:27.674240 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" path="/var/lib/kubelet/pods/6ffc5c9e-1a15-4d4c-9762-18fc84808b8f/volumes" Nov 02 06:43:27.674971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:27.674950 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" path="/var/lib/kubelet/pods/b575a0a5-5c30-4169-bc8b-e2aeb933d15f/volumes" Nov 02 06:43:27.675610 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:27.675591 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" path="/var/lib/kubelet/pods/c41dade0-68ae-4d60-99de-33151bf4b2b8/volumes" Nov 02 06:43:27.705417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:27.705389 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:30.507835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.507801 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:30.862144 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.856532 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:30.886293 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.886252 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr"] Nov 02 06:43:30.887855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.887830 5025 cpu_manager.go:414] "containerMap: removing container" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" Nov 02 06:43:30.888031 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.888015 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" Nov 02 06:43:30.888205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.888180 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerName="istio-proxy" Nov 02 06:43:30.888316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.888301 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerName="istio-proxy" Nov 02 06:43:30.888432 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.888422 5025 cpu_manager.go:414] "containerMap: removing container" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" Nov 02 06:43:30.888523 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.888503 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" Nov 02 06:43:30.888814 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.888795 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41dade0-68ae-4d60-99de-33151bf4b2b8" containerName="istio-proxy" Nov 02 06:43:30.888933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.888922 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b575a0a5-5c30-4169-bc8b-e2aeb933d15f" containerName="istio-proxy" Nov 02 06:43:30.889044 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.889026 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffc5c9e-1a15-4d4c-9762-18fc84808b8f" containerName="istio-proxy" Nov 02 06:43:30.895316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.895293 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:43:30.965515 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.965477 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-70898f\"" Nov 02 06:43:30.986146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.977973 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-70898f-dockercfg-4t9gt\"" Nov 02 06:43:31.024818 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:30.988610 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt668\" (UniqueName: \"kubernetes.io/projected/8bd5a095-dd31-4987-bf49-4fd9616a022c-kube-api-access-zt668\") pod \"payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr\" (UID: \"8bd5a095-dd31-4987-bf49-4fd9616a022c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:43:31.059936 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:31.025457 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd5a095-dd31-4987-bf49-4fd9616a022c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr\" (UID: \"8bd5a095-dd31-4987-bf49-4fd9616a022c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:43:31.115147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:31.093209 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr"] Nov 02 06:43:31.222096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:31.214946 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd5a095-dd31-4987-bf49-4fd9616a022c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr\" (UID: \"8bd5a095-dd31-4987-bf49-4fd9616a022c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:43:31.222096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:31.215188 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt668\" (UniqueName: \"kubernetes.io/projected/8bd5a095-dd31-4987-bf49-4fd9616a022c-kube-api-access-zt668\") pod \"payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr\" (UID: \"8bd5a095-dd31-4987-bf49-4fd9616a022c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:43:31.234142 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:31.230274 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd5a095-dd31-4987-bf49-4fd9616a022c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr\" (UID: \"8bd5a095-dd31-4987-bf49-4fd9616a022c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:43:31.350102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:31.343512 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt668\" (UniqueName: \"kubernetes.io/projected/8bd5a095-dd31-4987-bf49-4fd9616a022c-kube-api-access-zt668\") pod \"payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr\" (UID: \"8bd5a095-dd31-4987-bf49-4fd9616a022c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:43:31.598449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:31.598416 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:43:31.973674 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:43:31.973644 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd5a095_dd31_4987_bf49_4fd9616a022c.slice/crio-7bd5381b1ae7fe17a8a95aae4ce30351ddcdc3da713d4d2491ed193a1503b85b WatchSource:0}: Error finding container 7bd5381b1ae7fe17a8a95aae4ce30351ddcdc3da713d4d2491ed193a1503b85b: Status 404 returned error can't find the container with id 7bd5381b1ae7fe17a8a95aae4ce30351ddcdc3da713d4d2491ed193a1503b85b Nov 02 06:43:31.974135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:31.974109 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr"] Nov 02 06:43:32.564211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:32.564172 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" event={"ID":"8bd5a095-dd31-4987-bf49-4fd9616a022c","Type":"ContainerStarted","Data":"17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c"} Nov 02 06:43:32.564401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:32.564234 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" event={"ID":"8bd5a095-dd31-4987-bf49-4fd9616a022c","Type":"ContainerStarted","Data":"7bd5381b1ae7fe17a8a95aae4ce30351ddcdc3da713d4d2491ed193a1503b85b"} Nov 02 06:43:32.564401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:32.564264 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:43:32.709116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:32.703260 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" podStartSLOduration=2.703234832 podStartE2EDuration="2.703234832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:43:32.674994144 +0000 UTC m=+1589.123370919" watchObservedRunningTime="2026-11-02 06:43:32.703234832 +0000 UTC m=+1589.151611602" Nov 02 06:43:32.788302 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:43:32.787748 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:43:35.475005 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:35.474959 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:40.064487 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:40.064324 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:43.575914 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:43.575884 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:43:44.806981 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:44.806944 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:49.097585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:49.097551 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm"] Nov 02 06:43:49.098096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:49.097769 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" containerID="cri-o://8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9" gracePeriod=30 Nov 02 06:43:49.145070 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:49.145038 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p"] Nov 02 06:43:49.145276 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:49.145250 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" containerID="cri-o://9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce" gracePeriod=30 Nov 02 06:43:49.668097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:49.664032 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:50.348833 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:50.348797 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:50.502741 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:50.502708 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh"] Nov 02 06:43:50.502917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:50.502874 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" containerID="cri-o://e04e6214d96ab6a7f45291dd9ab5c5db36fef301842a4f4e6fcb69a054fbcf2a" gracePeriod=30 Nov 02 06:43:50.550671 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:50.550635 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc"] Nov 02 06:43:50.550910 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:50.550886 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" containerID="cri-o://3ceca20941b1cf883e559f096c46a85f6104d2fdad8a5d30c7146296636abbc5" gracePeriod=30 Nov 02 06:43:50.554973 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:50.554951 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9"] Nov 02 06:43:50.555200 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:50.555177 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" podUID="3eca8a46-979f-4e66-bcc2-d10108bf9305" containerName="istio-proxy" containerID="cri-o://291690b6303db10fb5ddeafb879b74bf32d29555b316fcb8a217bedb803b195c" gracePeriod=30 Nov 02 06:43:50.796819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:50.796786 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:51.423536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.423500 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75"] Nov 02 06:43:51.454694 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.454668 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.520668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.520628 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fcs2\" (UniqueName: \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-kube-api-access-6fcs2\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.530523 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.530491 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-token\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.530705 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.530577 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-data\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.530705 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.530618 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-credential-socket\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.530877 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.530815 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-podinfo\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.530933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.530888 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-socket\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.530983 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.530960 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288d008c-009e-4543-ae54-4ef60cc18c1a-istiod-ca-cert\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.531038 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.531009 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-envoy\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.531137 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.531038 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-certs\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.531330 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.524361 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-6e2657-openshift-default-dockercfg-9qlsh\"" Nov 02 06:43:51.597099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.574973 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75"] Nov 02 06:43:51.625038 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.625001 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:51.638111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.635307 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fcs2\" (UniqueName: \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-kube-api-access-6fcs2\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.638111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.635372 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-token\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.638111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.635438 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-data\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.638111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.635493 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-credential-socket\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.638111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.635661 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-podinfo\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.638111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.635722 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-socket\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.638111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.635785 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288d008c-009e-4543-ae54-4ef60cc18c1a-istiod-ca-cert\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.638111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.636742 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-data\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.678791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.638952 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-envoy\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.678791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.639011 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-certs\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.678791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.639509 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-certs\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.678791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.639733 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-socket\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.678791 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.640336 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288d008c-009e-4543-ae54-4ef60cc18c1a-istiod-ca-cert\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.689102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.679828 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-podinfo\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.689102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.680151 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-credential-socket\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.725582 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.725540 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-envoy\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.744875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.744844 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-token\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:51.746729 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:51.746674 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fcs2\" (UniqueName: \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-kube-api-access-6fcs2\") pod \"e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:52.047376 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:52.047286 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:52.295362 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:52.288356 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:52.475981 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:43:52.475954 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod288d008c_009e_4543_ae54_4ef60cc18c1a.slice/crio-4b832ccd126454d1841433b2e70e7b30fb838549e2e635a888a5ecda7826aaed WatchSource:0}: Error finding container 4b832ccd126454d1841433b2e70e7b30fb838549e2e635a888a5ecda7826aaed: Status 404 returned error can't find the container with id 4b832ccd126454d1841433b2e70e7b30fb838549e2e635a888a5ecda7826aaed Nov 02 06:43:52.476874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:52.476852 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75"] Nov 02 06:43:52.478687 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:52.478652 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:52.478793 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:52.478726 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:52.478793 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:52.478773 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:43:52.828866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:52.828750 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" event={"ID":"288d008c-009e-4543-ae54-4ef60cc18c1a","Type":"ContainerStarted","Data":"944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c"} Nov 02 06:43:52.828866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:52.828832 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" event={"ID":"288d008c-009e-4543-ae54-4ef60cc18c1a","Type":"ContainerStarted","Data":"4b832ccd126454d1841433b2e70e7b30fb838549e2e635a888a5ecda7826aaed"} Nov 02 06:43:52.945880 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:52.945817 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" podStartSLOduration=1.945797673 podStartE2EDuration="1.945797673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:43:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:43:52.944339468 +0000 UTC m=+1609.392716245" watchObservedRunningTime="2026-11-02 06:43:52.945797673 +0000 UTC m=+1609.394174429" Nov 02 06:43:53.047842 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:53.047812 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:53.063998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:53.063469 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:53.835343 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:53.835309 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:53.836397 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:53.836374 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:43:54.458220 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.458193 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:43:54.516635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.515809 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-kube-api-access-944zh\" (UniqueName: \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-kube-api-access-944zh\") pod \"4c758e46-4a47-4aa9-abee-798b5c3c0111\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " Nov 02 06:43:54.516635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.515942 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c758e46-4a47-4aa9-abee-798b5c3c0111-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c758e46-4a47-4aa9-abee-798b5c3c0111-istiod-ca-cert\") pod \"4c758e46-4a47-4aa9-abee-798b5c3c0111\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " Nov 02 06:43:54.516635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.515992 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-podinfo\") pod \"4c758e46-4a47-4aa9-abee-798b5c3c0111\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " Nov 02 06:43:54.516635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.516140 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-data\") pod \"4c758e46-4a47-4aa9-abee-798b5c3c0111\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " Nov 02 06:43:54.516635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.516188 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-certs\") pod \"4c758e46-4a47-4aa9-abee-798b5c3c0111\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " Nov 02 06:43:54.516635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.516252 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-socket\") pod \"4c758e46-4a47-4aa9-abee-798b5c3c0111\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " Nov 02 06:43:54.516635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.516314 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-token\" (UniqueName: \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-token\") pod \"4c758e46-4a47-4aa9-abee-798b5c3c0111\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " Nov 02 06:43:54.516635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.516410 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-credential-socket\") pod \"4c758e46-4a47-4aa9-abee-798b5c3c0111\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " Nov 02 06:43:54.516635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.516451 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-envoy\") pod \"4c758e46-4a47-4aa9-abee-798b5c3c0111\" (UID: \"4c758e46-4a47-4aa9-abee-798b5c3c0111\") " Nov 02 06:43:54.518291 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.517959 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-data" pod "4c758e46-4a47-4aa9-abee-798b5c3c0111" (UID: "4c758e46-4a47-4aa9-abee-798b5c3c0111"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.518779 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.518551 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c758e46-4a47-4aa9-abee-798b5c3c0111-istiod-ca-cert" pod "4c758e46-4a47-4aa9-abee-798b5c3c0111" (UID: "4c758e46-4a47-4aa9-abee-798b5c3c0111"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:54.519698 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.519434 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c758e46-4a47-4aa9-abee-798b5c3c0111-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.519698 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.519460 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.522494 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.522034 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-socket" pod "4c758e46-4a47-4aa9-abee-798b5c3c0111" (UID: "4c758e46-4a47-4aa9-abee-798b5c3c0111"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.522494 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.522244 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-certs" pod "4c758e46-4a47-4aa9-abee-798b5c3c0111" (UID: "4c758e46-4a47-4aa9-abee-798b5c3c0111"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.523302 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.523205 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-credential-socket" pod "4c758e46-4a47-4aa9-abee-798b5c3c0111" (UID: "4c758e46-4a47-4aa9-abee-798b5c3c0111"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.530168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.530123 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-token" pod "4c758e46-4a47-4aa9-abee-798b5c3c0111" (UID: "4c758e46-4a47-4aa9-abee-798b5c3c0111"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.532478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.532443 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-envoy" pod "4c758e46-4a47-4aa9-abee-798b5c3c0111" (UID: "4c758e46-4a47-4aa9-abee-798b5c3c0111"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.533475 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.533413 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:43:54.554116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.548673 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-kube-api-access-944zh" pod "4c758e46-4a47-4aa9-abee-798b5c3c0111" (UID: "4c758e46-4a47-4aa9-abee-798b5c3c0111"). InnerVolumeSpecName "kube-api-access-944zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.555285 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.555252 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-podinfo" pod "4c758e46-4a47-4aa9-abee-798b5c3c0111" (UID: "4c758e46-4a47-4aa9-abee-798b5c3c0111"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.632802 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-credential-socket\") pod \"d047f344-8af2-4098-847c-a8e1447e7bc8\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.632941 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-envoy\") pod \"d047f344-8af2-4098-847c-a8e1447e7bc8\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.633018 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-token\" (UniqueName: \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-token\") pod \"d047f344-8af2-4098-847c-a8e1447e7bc8\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.633148 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-socket\") pod \"d047f344-8af2-4098-847c-a8e1447e7bc8\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.633179 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-certs\") pod \"d047f344-8af2-4098-847c-a8e1447e7bc8\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.633214 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d047f344-8af2-4098-847c-a8e1447e7bc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d047f344-8af2-4098-847c-a8e1447e7bc8-istiod-ca-cert\") pod \"d047f344-8af2-4098-847c-a8e1447e7bc8\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.633263 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-kube-api-access-n6mf6\" (UniqueName: \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-kube-api-access-n6mf6\") pod \"d047f344-8af2-4098-847c-a8e1447e7bc8\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.633300 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-data\") pod \"d047f344-8af2-4098-847c-a8e1447e7bc8\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.633327 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-podinfo\") pod \"d047f344-8af2-4098-847c-a8e1447e7bc8\" (UID: \"d047f344-8af2-4098-847c-a8e1447e7bc8\") " Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634159 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-credential-socket" pod "d047f344-8af2-4098-847c-a8e1447e7bc8" (UID: "d047f344-8af2-4098-847c-a8e1447e7bc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634262 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634289 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634302 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634315 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634327 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634340 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-944zh\" (UniqueName: \"kubernetes.io/projected/4c758e46-4a47-4aa9-abee-798b5c3c0111-kube-api-access-944zh\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634354 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c758e46-4a47-4aa9-abee-798b5c3c0111-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634389 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-data" pod "d047f344-8af2-4098-847c-a8e1447e7bc8" (UID: "d047f344-8af2-4098-847c-a8e1447e7bc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634463 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d047f344-8af2-4098-847c-a8e1447e7bc8-istiod-ca-cert" pod "d047f344-8af2-4098-847c-a8e1447e7bc8" (UID: "d047f344-8af2-4098-847c-a8e1447e7bc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634727 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-certs" pod "d047f344-8af2-4098-847c-a8e1447e7bc8" (UID: "d047f344-8af2-4098-847c-a8e1447e7bc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.634917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.634776 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-socket" pod "d047f344-8af2-4098-847c-a8e1447e7bc8" (UID: "d047f344-8af2-4098-847c-a8e1447e7bc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.639250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.638228 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-podinfo" pod "d047f344-8af2-4098-847c-a8e1447e7bc8" (UID: "d047f344-8af2-4098-847c-a8e1447e7bc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:54.639250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.638957 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-kube-api-access-n6mf6" pod "d047f344-8af2-4098-847c-a8e1447e7bc8" (UID: "d047f344-8af2-4098-847c-a8e1447e7bc8"). InnerVolumeSpecName "kube-api-access-n6mf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.639250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.639151 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-token" pod "d047f344-8af2-4098-847c-a8e1447e7bc8" (UID: "d047f344-8af2-4098-847c-a8e1447e7bc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:54.640801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.640656 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-envoy" pod "d047f344-8af2-4098-847c-a8e1447e7bc8" (UID: "d047f344-8af2-4098-847c-a8e1447e7bc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:54.738904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.738046 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.738904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.738122 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.738904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.738139 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.738904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.738153 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.738904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.738167 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.738904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.738181 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d047f344-8af2-4098-847c-a8e1447e7bc8-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.738904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.738194 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6mf6\" (UniqueName: \"kubernetes.io/projected/d047f344-8af2-4098-847c-a8e1447e7bc8-kube-api-access-n6mf6\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.738904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.738207 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.738904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.738220 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d047f344-8af2-4098-847c-a8e1447e7bc8-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:54.843963 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.843925 5025 generic.go:356] "Generic (PLEG): container finished" podID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerID="9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce" exitCode=0 Nov 02 06:43:54.844418 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.843972 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" Nov 02 06:43:54.844418 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.844005 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" event={"ID":"d047f344-8af2-4098-847c-a8e1447e7bc8","Type":"ContainerDied","Data":"9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce"} Nov 02 06:43:54.844418 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.844059 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p" event={"ID":"d047f344-8af2-4098-847c-a8e1447e7bc8","Type":"ContainerDied","Data":"b1105775f01b027e9b7800e2557ab7af2b19e350471362876e44fa443e8bffd9"} Nov 02 06:43:54.844418 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.844099 5025 scope.go:122] "RemoveContainer" containerID="9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce" Nov 02 06:43:54.845704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.845677 5025 generic.go:356] "Generic (PLEG): container finished" podID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerID="8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9" exitCode=0 Nov 02 06:43:54.845851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.845746 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" Nov 02 06:43:54.845851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.845745 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" event={"ID":"4c758e46-4a47-4aa9-abee-798b5c3c0111","Type":"ContainerDied","Data":"8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9"} Nov 02 06:43:54.845970 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.845906 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm" event={"ID":"4c758e46-4a47-4aa9-abee-798b5c3c0111","Type":"ContainerDied","Data":"01e1e056a56b4f438864d204455fe448cea866ff4024e69f285b78b5818a3396"} Nov 02 06:43:54.872210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.870654 5025 scope.go:122] "RemoveContainer" containerID="9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce" Nov 02 06:43:54.872210 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:43:54.871032 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce\": container with ID starting with 9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce not found: ID does not exist" containerID="9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce" Nov 02 06:43:54.872210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.871061 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce"} err="failed to get container status \"9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce\": rpc error: code = NotFound desc = could not find container \"9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce\": container with ID starting with 9267ad1cae315b9a7357e0aea03c298983f62c8d647025c5e68bc4d51b0491ce not found: ID does not exist" Nov 02 06:43:54.872210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.871124 5025 scope.go:122] "RemoveContainer" containerID="8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9" Nov 02 06:43:54.882630 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.882608 5025 scope.go:122] "RemoveContainer" containerID="8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9" Nov 02 06:43:54.882893 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:43:54.882875 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9\": container with ID starting with 8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9 not found: ID does not exist" containerID="8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9" Nov 02 06:43:54.882956 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:54.882900 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9"} err="failed to get container status \"8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9\": rpc error: code = NotFound desc = could not find container \"8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9\": container with ID starting with 8165f784acacf81075f65789d77f09b504ac0793e19f63e497e4d7311bef4cf9 not found: ID does not exist" Nov 02 06:43:55.025291 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.025200 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:55.059249 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.051139 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p"] Nov 02 06:43:55.070102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.065439 5025 patch_prober.go:29] interesting pod/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:43:55.070102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.065501 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:43:55.111104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.110559 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-8kv5p"] Nov 02 06:43:55.151098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.145247 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm"] Nov 02 06:43:55.164941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.164908 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-5sjrm"] Nov 02 06:43:55.450421 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.450390 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:55.720427 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.709432 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" path="/var/lib/kubelet/pods/4c758e46-4a47-4aa9-abee-798b5c3c0111/volumes" Nov 02 06:43:55.720427 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.710302 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" path="/var/lib/kubelet/pods/d047f344-8af2-4098-847c-a8e1447e7bc8/volumes" Nov 02 06:43:55.858250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.858213 5025 generic.go:356] "Generic (PLEG): container finished" podID="3eca8a46-979f-4e66-bcc2-d10108bf9305" containerID="291690b6303db10fb5ddeafb879b74bf32d29555b316fcb8a217bedb803b195c" exitCode=0 Nov 02 06:43:55.858701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.858292 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" event={"ID":"3eca8a46-979f-4e66-bcc2-d10108bf9305","Type":"ContainerDied","Data":"291690b6303db10fb5ddeafb879b74bf32d29555b316fcb8a217bedb803b195c"} Nov 02 06:43:55.861207 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.861168 5025 generic.go:356] "Generic (PLEG): container finished" podID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerID="3ceca20941b1cf883e559f096c46a85f6104d2fdad8a5d30c7146296636abbc5" exitCode=0 Nov 02 06:43:55.861354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.861247 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" event={"ID":"18e926c7-f29a-4ad2-b404-b1920c3a15d7","Type":"ContainerDied","Data":"3ceca20941b1cf883e559f096c46a85f6104d2fdad8a5d30c7146296636abbc5"} Nov 02 06:43:55.864561 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.864289 5025 generic.go:356] "Generic (PLEG): container finished" podID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerID="e04e6214d96ab6a7f45291dd9ab5c5db36fef301842a4f4e6fcb69a054fbcf2a" exitCode=0 Nov 02 06:43:55.864561 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.864349 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" event={"ID":"b9dab3c5-b0ed-4b9c-954f-8350b87248bd","Type":"ContainerDied","Data":"e04e6214d96ab6a7f45291dd9ab5c5db36fef301842a4f4e6fcb69a054fbcf2a"} Nov 02 06:43:55.963179 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:55.963151 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:43:56.026022 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.025998 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:43:56.030541 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.030522 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:43:56.074004 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.073977 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-data\") pod \"3eca8a46-979f-4e66-bcc2-d10108bf9305\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " Nov 02 06:43:56.074172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074020 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-token\" (UniqueName: \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-token\") pod \"3eca8a46-979f-4e66-bcc2-d10108bf9305\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " Nov 02 06:43:56.074172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074062 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-certs\") pod \"3eca8a46-979f-4e66-bcc2-d10108bf9305\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " Nov 02 06:43:56.074172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074149 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-podinfo\") pod \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " Nov 02 06:43:56.074172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074169 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eca8a46-979f-4e66-bcc2-d10108bf9305-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eca8a46-979f-4e66-bcc2-d10108bf9305-istiod-ca-cert\") pod \"3eca8a46-979f-4e66-bcc2-d10108bf9305\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " Nov 02 06:43:56.074399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074194 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-socket\") pod \"3eca8a46-979f-4e66-bcc2-d10108bf9305\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " Nov 02 06:43:56.074399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074292 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-data" pod "3eca8a46-979f-4e66-bcc2-d10108bf9305" (UID: "3eca8a46-979f-4e66-bcc2-d10108bf9305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.074399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074317 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-socket\") pod \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " Nov 02 06:43:56.074399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074364 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-kube-api-access-g9sl6\" (UniqueName: \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-kube-api-access-g9sl6\") pod \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " Nov 02 06:43:56.074611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074450 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-socket\") pod \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " Nov 02 06:43:56.074611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074477 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-credential-socket\") pod \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " Nov 02 06:43:56.074611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074484 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-socket" pod "3eca8a46-979f-4e66-bcc2-d10108bf9305" (UID: "3eca8a46-979f-4e66-bcc2-d10108bf9305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.074611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074525 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istiod-ca-cert\") pod \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " Nov 02 06:43:56.074611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074539 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-certs" pod "3eca8a46-979f-4e66-bcc2-d10108bf9305" (UID: "3eca8a46-979f-4e66-bcc2-d10108bf9305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.074611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074550 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-socket" pod "b9dab3c5-b0ed-4b9c-954f-8350b87248bd" (UID: "b9dab3c5-b0ed-4b9c-954f-8350b87248bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.074611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074556 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-certs\") pod \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " Nov 02 06:43:56.074611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074580 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eca8a46-979f-4e66-bcc2-d10108bf9305-istiod-ca-cert" pod "3eca8a46-979f-4e66-bcc2-d10108bf9305" (UID: "3eca8a46-979f-4e66-bcc2-d10108bf9305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:56.074995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074621 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istiod-ca-cert\") pod \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " Nov 02 06:43:56.074995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074682 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-token\" (UniqueName: \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-token\") pod \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " Nov 02 06:43:56.074995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074738 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-data\") pod \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " Nov 02 06:43:56.074995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074765 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-envoy\") pod \"3eca8a46-979f-4e66-bcc2-d10108bf9305\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " Nov 02 06:43:56.074995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074766 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-socket" pod "18e926c7-f29a-4ad2-b404-b1920c3a15d7" (UID: "18e926c7-f29a-4ad2-b404-b1920c3a15d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.074995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074805 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-podinfo\") pod \"3eca8a46-979f-4e66-bcc2-d10108bf9305\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " Nov 02 06:43:56.074995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074851 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-certs\") pod \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " Nov 02 06:43:56.074995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074876 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-credential-socket" pod "18e926c7-f29a-4ad2-b404-b1920c3a15d7" (UID: "18e926c7-f29a-4ad2-b404-b1920c3a15d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.074995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074933 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istiod-ca-cert" pod "b9dab3c5-b0ed-4b9c-954f-8350b87248bd" (UID: "b9dab3c5-b0ed-4b9c-954f-8350b87248bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:56.074995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.074957 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-envoy\") pod \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " Nov 02 06:43:56.075554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075010 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-credential-socket\") pod \"3eca8a46-979f-4e66-bcc2-d10108bf9305\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " Nov 02 06:43:56.075554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075062 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-kube-api-access-cbgs2\" (UniqueName: \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-kube-api-access-cbgs2\") pod \"3eca8a46-979f-4e66-bcc2-d10108bf9305\" (UID: \"3eca8a46-979f-4e66-bcc2-d10108bf9305\") " Nov 02 06:43:56.075554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075125 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-podinfo\") pod \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " Nov 02 06:43:56.075554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075391 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-certs" pod "b9dab3c5-b0ed-4b9c-954f-8350b87248bd" (UID: "b9dab3c5-b0ed-4b9c-954f-8350b87248bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.075744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075642 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-credential-socket\") pod \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " Nov 02 06:43:56.075799 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075658 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-credential-socket" pod "3eca8a46-979f-4e66-bcc2-d10108bf9305" (UID: "3eca8a46-979f-4e66-bcc2-d10108bf9305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.075799 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075763 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-data\") pod \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " Nov 02 06:43:56.075799 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075714 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-credential-socket" pod "b9dab3c5-b0ed-4b9c-954f-8350b87248bd" (UID: "b9dab3c5-b0ed-4b9c-954f-8350b87248bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.075799 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075749 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istiod-ca-cert" pod "18e926c7-f29a-4ad2-b404-b1920c3a15d7" (UID: "18e926c7-f29a-4ad2-b404-b1920c3a15d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:43:56.075991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075813 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-kube-api-access-2zfv9\" (UniqueName: \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-kube-api-access-2zfv9\") pod \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " Nov 02 06:43:56.075991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075860 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-envoy\") pod \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\" (UID: \"b9dab3c5-b0ed-4b9c-954f-8350b87248bd\") " Nov 02 06:43:56.075991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.075896 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-token\" (UniqueName: \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-token\") pod \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\" (UID: \"18e926c7-f29a-4ad2-b404-b1920c3a15d7\") " Nov 02 06:43:56.076265 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076024 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-data" pod "18e926c7-f29a-4ad2-b404-b1920c3a15d7" (UID: "18e926c7-f29a-4ad2-b404-b1920c3a15d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.076265 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076173 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-certs" pod "18e926c7-f29a-4ad2-b404-b1920c3a15d7" (UID: "18e926c7-f29a-4ad2-b404-b1920c3a15d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.076375 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076296 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-data" pod "b9dab3c5-b0ed-4b9c-954f-8350b87248bd" (UID: "b9dab3c5-b0ed-4b9c-954f-8350b87248bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076860 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076889 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076904 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076918 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076932 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076946 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076960 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076973 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.076987 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.077000 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.077012 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.077025 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.077038 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eca8a46-979f-4e66-bcc2-d10108bf9305-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.077090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.077044 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-token" pod "3eca8a46-979f-4e66-bcc2-d10108bf9305" (UID: "3eca8a46-979f-4e66-bcc2-d10108bf9305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:56.078814 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.077052 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.078814 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.077137 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.078814 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.077688 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-podinfo" pod "18e926c7-f29a-4ad2-b404-b1920c3a15d7" (UID: "18e926c7-f29a-4ad2-b404-b1920c3a15d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:56.078814 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.078112 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-token" pod "b9dab3c5-b0ed-4b9c-954f-8350b87248bd" (UID: "b9dab3c5-b0ed-4b9c-954f-8350b87248bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:56.078814 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.078437 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-kube-api-access-g9sl6" pod "18e926c7-f29a-4ad2-b404-b1920c3a15d7" (UID: "18e926c7-f29a-4ad2-b404-b1920c3a15d7"). InnerVolumeSpecName "kube-api-access-g9sl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:56.081360 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.081324 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-token" pod "18e926c7-f29a-4ad2-b404-b1920c3a15d7" (UID: "18e926c7-f29a-4ad2-b404-b1920c3a15d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:56.081484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.081459 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-envoy" pod "b9dab3c5-b0ed-4b9c-954f-8350b87248bd" (UID: "b9dab3c5-b0ed-4b9c-954f-8350b87248bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.081546 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.081492 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-kube-api-access-2zfv9" pod "b9dab3c5-b0ed-4b9c-954f-8350b87248bd" (UID: "b9dab3c5-b0ed-4b9c-954f-8350b87248bd"). InnerVolumeSpecName "kube-api-access-2zfv9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:56.081888 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.081859 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-podinfo" pod "b9dab3c5-b0ed-4b9c-954f-8350b87248bd" (UID: "b9dab3c5-b0ed-4b9c-954f-8350b87248bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:56.081989 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.081900 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-podinfo" pod "3eca8a46-979f-4e66-bcc2-d10108bf9305" (UID: "3eca8a46-979f-4e66-bcc2-d10108bf9305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:43:56.081989 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.081918 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-kube-api-access-cbgs2" pod "3eca8a46-979f-4e66-bcc2-d10108bf9305" (UID: "3eca8a46-979f-4e66-bcc2-d10108bf9305"). InnerVolumeSpecName "kube-api-access-cbgs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:43:56.097019 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.094249 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-envoy" pod "3eca8a46-979f-4e66-bcc2-d10108bf9305" (UID: "3eca8a46-979f-4e66-bcc2-d10108bf9305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.111620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.111568 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-envoy" pod "18e926c7-f29a-4ad2-b404-b1920c3a15d7" (UID: "18e926c7-f29a-4ad2-b404-b1920c3a15d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:43:56.178899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.178869 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.179036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.178903 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.179036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.178916 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.179036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.178924 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.179036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.178934 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbgs2\" (UniqueName: \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-kube-api-access-cbgs2\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.179036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.178942 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.179036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.178950 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zfv9\" (UniqueName: \"kubernetes.io/projected/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-kube-api-access-2zfv9\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.179036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.178964 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9dab3c5-b0ed-4b9c-954f-8350b87248bd-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.179036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.178978 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.179036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.178991 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eca8a46-979f-4e66-bcc2-d10108bf9305-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.179036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.179006 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e926c7-f29a-4ad2-b404-b1920c3a15d7-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.179036 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.179018 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9sl6\" (UniqueName: \"kubernetes.io/projected/18e926c7-f29a-4ad2-b404-b1920c3a15d7-kube-api-access-g9sl6\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:43:56.344284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.343932 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:56.624214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.621946 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:56.813389 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.813355 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:56.872551 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.872521 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" Nov 02 06:43:56.873010 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.872556 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc" event={"ID":"18e926c7-f29a-4ad2-b404-b1920c3a15d7","Type":"ContainerDied","Data":"a1465ba52c391fd51babeb1e1cf5d58b42bf23ac5c8b2cd7a4c351ff54418741"} Nov 02 06:43:56.873010 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.872624 5025 scope.go:122] "RemoveContainer" containerID="3ceca20941b1cf883e559f096c46a85f6104d2fdad8a5d30c7146296636abbc5" Nov 02 06:43:56.874627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.874562 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" Nov 02 06:43:56.874627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.874573 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh" event={"ID":"b9dab3c5-b0ed-4b9c-954f-8350b87248bd","Type":"ContainerDied","Data":"0da516ec749a744b2c113032e003aead1d6bf3ffab67bc5d50d8df916fd49d53"} Nov 02 06:43:56.882497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.882408 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" Nov 02 06:43:56.882497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.882446 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9" event={"ID":"3eca8a46-979f-4e66-bcc2-d10108bf9305","Type":"ContainerDied","Data":"59d6d13340181a4c3e6cd98b7e7a44be4138e50bc37d058b2cab2e3bab51b85a"} Nov 02 06:43:56.913097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.904268 5025 scope.go:122] "RemoveContainer" containerID="e04e6214d96ab6a7f45291dd9ab5c5db36fef301842a4f4e6fcb69a054fbcf2a" Nov 02 06:43:56.925961 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:56.925869 5025 scope.go:122] "RemoveContainer" containerID="291690b6303db10fb5ddeafb879b74bf32d29555b316fcb8a217bedb803b195c" Nov 02 06:43:57.036433 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:57.036400 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc"] Nov 02 06:43:57.102101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:57.083602 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-mv6mc"] Nov 02 06:43:57.167359 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:57.167315 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:43:57.168380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:57.168355 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9"] Nov 02 06:43:57.171744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:57.171723 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-f2ns9"] Nov 02 06:43:57.275670 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:57.275635 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh"] Nov 02 06:43:57.279668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:57.279643 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-tnwwh"] Nov 02 06:43:57.674934 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:57.674896 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" path="/var/lib/kubelet/pods/18e926c7-f29a-4ad2-b404-b1920c3a15d7/volumes" Nov 02 06:43:57.675826 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:57.675798 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eca8a46-979f-4e66-bcc2-d10108bf9305" path="/var/lib/kubelet/pods/3eca8a46-979f-4e66-bcc2-d10108bf9305/volumes" Nov 02 06:43:57.676636 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:43:57.676613 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" path="/var/lib/kubelet/pods/b9dab3c5-b0ed-4b9c-954f-8350b87248bd/volumes" Nov 02 06:44:00.329150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:00.327887 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:01.957233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:01.957194 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:02.164145 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:02.159391 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:02.730938 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:02.730905 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:44:03.658653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:03.658617 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:04.053253 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:04.053216 5025 scope.go:122] "RemoveContainer" containerID="b2c821b8ed0976c74b40d1bd016375863f13015530597e24e392881b41bb895b" Nov 02 06:44:04.966788 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:04.966757 5025 generic.go:356] "Generic (PLEG): container finished" podID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerID="36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f" exitCode=0 Nov 02 06:44:04.967227 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:04.966794 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" event={"ID":"a3992d0d-ee6b-48dc-a41a-dc2302cc0977","Type":"ContainerDied","Data":"36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f"} Nov 02 06:44:04.967299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:04.967269 5025 scope.go:122] "RemoveContainer" containerID="36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f" Nov 02 06:44:04.968888 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:04.968858 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:44:04.968953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:04.968929 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:44:04.968994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:04.968961 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:44:05.604144 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:05.603757 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:05.976487 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:05.976448 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" event={"ID":"a3992d0d-ee6b-48dc-a41a-dc2302cc0977","Type":"ContainerStarted","Data":"27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef"} Nov 02 06:44:06.960663 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:06.960626 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:44:06.961811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:06.961773 5025 patch_prober.go:29] interesting pod/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:44:06.961913 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:06.961837 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 06:44:07.959798 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:07.959763 5025 patch_prober.go:29] interesting pod/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:44:07.960194 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:07.959815 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 06:44:08.960026 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:08.959991 5025 patch_prober.go:29] interesting pod/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:44:08.960427 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:08.960044 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 06:44:09.959874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:09.959843 5025 patch_prober.go:29] interesting pod/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:44:09.960057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:09.959889 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 06:44:10.113288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:10.113249 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:10.305849 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:10.305731 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:10.960064 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:10.960028 5025 patch_prober.go:29] interesting pod/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" start-of-body= Nov 02 06:44:10.960064 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:10.960093 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.95:15021/healthz/ready\": dial tcp 10.133.0.95:15021: connect: connection refused" Nov 02 06:44:11.838205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:11.838172 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:44:11.964919 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:11.964886 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:44:12.043480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:12.043454 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:44:14.847126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:14.845879 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:19.453100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:19.451838 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:22.676267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:22.676220 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:24.238110 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:24.232426 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:28.942477 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:28.942443 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:29.127959 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:29.127919 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:29.315011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:29.314913 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:32.732576 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:32.732529 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:44:33.883118 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:33.878997 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:34.724256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:34.724222 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj"] Nov 02 06:44:34.724474 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:34.724436 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" podUID="62c5c605-5c52-4c00-9d51-701c40c993d5" containerName="istio-proxy" containerID="cri-o://5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676" gracePeriod=30 Nov 02 06:44:38.584276 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:38.584222 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:39.999308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:39.999250 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:44:40.128139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128108 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-certs\") pod \"62c5c605-5c52-4c00-9d51-701c40c993d5\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " Nov 02 06:44:40.128339 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128155 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-kube-api-access-gfv2g\" (UniqueName: \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-kube-api-access-gfv2g\") pod \"62c5c605-5c52-4c00-9d51-701c40c993d5\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " Nov 02 06:44:40.128339 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128196 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-socket\") pod \"62c5c605-5c52-4c00-9d51-701c40c993d5\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " Nov 02 06:44:40.128339 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128329 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-podinfo\") pod \"62c5c605-5c52-4c00-9d51-701c40c993d5\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " Nov 02 06:44:40.128451 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128381 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-envoy\") pod \"62c5c605-5c52-4c00-9d51-701c40c993d5\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " Nov 02 06:44:40.128451 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128382 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-certs" pod "62c5c605-5c52-4c00-9d51-701c40c993d5" (UID: "62c5c605-5c52-4c00-9d51-701c40c993d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:40.128451 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128413 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62c5c605-5c52-4c00-9d51-701c40c993d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c5c605-5c52-4c00-9d51-701c40c993d5-istiod-ca-cert\") pod \"62c5c605-5c52-4c00-9d51-701c40c993d5\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " Nov 02 06:44:40.128597 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128581 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-token\" (UniqueName: \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-token\") pod \"62c5c605-5c52-4c00-9d51-701c40c993d5\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " Nov 02 06:44:40.128658 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128606 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-socket" pod "62c5c605-5c52-4c00-9d51-701c40c993d5" (UID: "62c5c605-5c52-4c00-9d51-701c40c993d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:40.128658 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128630 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-credential-socket\") pod \"62c5c605-5c52-4c00-9d51-701c40c993d5\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " Nov 02 06:44:40.128767 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128709 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-data\") pod \"62c5c605-5c52-4c00-9d51-701c40c993d5\" (UID: \"62c5c605-5c52-4c00-9d51-701c40c993d5\") " Nov 02 06:44:40.128991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.128849 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62c5c605-5c52-4c00-9d51-701c40c993d5-istiod-ca-cert" pod "62c5c605-5c52-4c00-9d51-701c40c993d5" (UID: "62c5c605-5c52-4c00-9d51-701c40c993d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:44:40.129193 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.129163 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-data" pod "62c5c605-5c52-4c00-9d51-701c40c993d5" (UID: "62c5c605-5c52-4c00-9d51-701c40c993d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:40.129193 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.129184 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-credential-socket" pod "62c5c605-5c52-4c00-9d51-701c40c993d5" (UID: "62c5c605-5c52-4c00-9d51-701c40c993d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:40.129571 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.129548 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:40.129660 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.129579 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:40.129660 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.129596 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:40.129660 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.129612 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:40.129660 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.129629 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62c5c605-5c52-4c00-9d51-701c40c993d5-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:40.130621 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.130591 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-podinfo" pod "62c5c605-5c52-4c00-9d51-701c40c993d5" (UID: "62c5c605-5c52-4c00-9d51-701c40c993d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:44:40.130705 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.130627 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-kube-api-access-gfv2g" pod "62c5c605-5c52-4c00-9d51-701c40c993d5" (UID: "62c5c605-5c52-4c00-9d51-701c40c993d5"). InnerVolumeSpecName "kube-api-access-gfv2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:44:40.130705 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.130649 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-envoy" pod "62c5c605-5c52-4c00-9d51-701c40c993d5" (UID: "62c5c605-5c52-4c00-9d51-701c40c993d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:44:40.130829 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.130811 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-token" pod "62c5c605-5c52-4c00-9d51-701c40c993d5" (UID: "62c5c605-5c52-4c00-9d51-701c40c993d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:44:40.231210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.231178 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:40.231210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.231203 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfv2g\" (UniqueName: \"kubernetes.io/projected/62c5c605-5c52-4c00-9d51-701c40c993d5-kube-api-access-gfv2g\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:40.231210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.231215 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:40.231436 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.231223 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62c5c605-5c52-4c00-9d51-701c40c993d5-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:40.355221 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.355133 5025 generic.go:356] "Generic (PLEG): container finished" podID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerID="efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244" exitCode=0 Nov 02 06:44:40.355384 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.355217 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" event={"ID":"8e296f80-34b3-44ad-b978-bcd9b99f19c1","Type":"ContainerDied","Data":"efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244"} Nov 02 06:44:40.355742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.355725 5025 scope.go:122] "RemoveContainer" containerID="efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244" Nov 02 06:44:40.356996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.356972 5025 generic.go:356] "Generic (PLEG): container finished" podID="62c5c605-5c52-4c00-9d51-701c40c993d5" containerID="5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676" exitCode=0 Nov 02 06:44:40.357194 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.357025 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" Nov 02 06:44:40.357194 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.357039 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" event={"ID":"62c5c605-5c52-4c00-9d51-701c40c993d5","Type":"ContainerDied","Data":"5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676"} Nov 02 06:44:40.357194 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.357093 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj" event={"ID":"62c5c605-5c52-4c00-9d51-701c40c993d5","Type":"ContainerDied","Data":"f5eca32d0b6c6ef754bebfb21d0a1650986a484092fde3e00bcdfea6a916f4d1"} Nov 02 06:44:40.357194 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.357124 5025 scope.go:122] "RemoveContainer" containerID="5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676" Nov 02 06:44:40.357991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.357921 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:44:40.358258 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.358053 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:44:40.358258 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.358213 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:44:40.376678 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.376660 5025 scope.go:122] "RemoveContainer" containerID="5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676" Nov 02 06:44:40.376968 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:40.376952 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676\": container with ID starting with 5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676 not found: ID does not exist" containerID="5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676" Nov 02 06:44:40.377011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.376975 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676"} err="failed to get container status \"5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676\": rpc error: code = NotFound desc = could not find container \"5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676\": container with ID starting with 5f57b2e2f231b2c285a53825440f791f936aa88f0ae68bd783016e9d4a7b8676 not found: ID does not exist" Nov 02 06:44:40.595021 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.594970 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj"] Nov 02 06:44:40.602737 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:40.602705 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-6qllj"] Nov 02 06:44:41.366702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:41.366669 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" event={"ID":"8e296f80-34b3-44ad-b978-bcd9b99f19c1","Type":"ContainerStarted","Data":"8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a"} Nov 02 06:44:41.671212 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:41.671178 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62c5c605-5c52-4c00-9d51-701c40c993d5" path="/var/lib/kubelet/pods/62c5c605-5c52-4c00-9d51-701c40c993d5/volumes" Nov 02 06:44:42.104156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.099457 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:44:42.104156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.099584 5025 patch_prober.go:29] interesting pod/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:44:42.104156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.099630 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 06:44:42.976966 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.976926 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k"] Nov 02 06:44:42.977882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.977861 5025 cpu_manager.go:414] "containerMap: removing container" podUID="62c5c605-5c52-4c00-9d51-701c40c993d5" containerName="istio-proxy" Nov 02 06:44:42.977882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.977882 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62c5c605-5c52-4c00-9d51-701c40c993d5" containerName="istio-proxy" Nov 02 06:44:42.978053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.977902 5025 cpu_manager.go:414] "containerMap: removing container" podUID="3eca8a46-979f-4e66-bcc2-d10108bf9305" containerName="istio-proxy" Nov 02 06:44:42.978053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.977910 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eca8a46-979f-4e66-bcc2-d10108bf9305" containerName="istio-proxy" Nov 02 06:44:42.978053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.977944 5025 cpu_manager.go:414] "containerMap: removing container" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" Nov 02 06:44:42.978053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.977952 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" Nov 02 06:44:42.978053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.977971 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" Nov 02 06:44:42.978053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.977977 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" Nov 02 06:44:42.978053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.977990 5025 cpu_manager.go:414] "containerMap: removing container" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" Nov 02 06:44:42.978053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.977996 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" Nov 02 06:44:42.978053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.978022 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" Nov 02 06:44:42.978053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.978029 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" Nov 02 06:44:42.978520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.978198 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e926c7-f29a-4ad2-b404-b1920c3a15d7" containerName="istio-proxy" Nov 02 06:44:42.978520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.978217 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9dab3c5-b0ed-4b9c-954f-8350b87248bd" containerName="istio-proxy" Nov 02 06:44:42.978520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.978227 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c758e46-4a47-4aa9-abee-798b5c3c0111" containerName="istio-proxy" Nov 02 06:44:42.978520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.978243 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d047f344-8af2-4098-847c-a8e1447e7bc8" containerName="istio-proxy" Nov 02 06:44:42.978520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.978256 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="62c5c605-5c52-4c00-9d51-701c40c993d5" containerName="istio-proxy" Nov 02 06:44:42.978520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.978266 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eca8a46-979f-4e66-bcc2-d10108bf9305" containerName="istio-proxy" Nov 02 06:44:42.981238 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:42.981217 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.034705 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.034673 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k"] Nov 02 06:44:43.078012 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.077977 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-70898f-serving-cert\"" Nov 02 06:44:43.082471 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.082433 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbxp\" (UniqueName: \"kubernetes.io/projected/6602a158-b770-4ce3-80d2-6c050ad2c591-kube-api-access-mlbxp\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.082599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.082558 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-maas-api-tls\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.082662 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.082642 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6602a158-b770-4ce3-80d2-6c050ad2c591-openshift-service-ca\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.082847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.082826 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-metrics-certs\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.108101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.101431 5025 patch_prober.go:29] interesting pod/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:44:43.108101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.101488 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 06:44:43.210115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.197035 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-maas-api-tls\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.210115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.208618 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6602a158-b770-4ce3-80d2-6c050ad2c591-openshift-service-ca\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.210115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.208835 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-metrics-certs\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.210115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.209028 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbxp\" (UniqueName: \"kubernetes.io/projected/6602a158-b770-4ce3-80d2-6c050ad2c591-kube-api-access-mlbxp\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.210115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.209920 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6602a158-b770-4ce3-80d2-6c050ad2c591-openshift-service-ca\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.210115 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:43.208297 5025 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-serving-cert: secret "maas-api-e2e-api-a-70898f-serving-cert" not found Nov 02 06:44:43.210115 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:43.210008 5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-maas-api-tls podName:6602a158-b770-4ce3-80d2-6c050ad2c591 nodeName:}" failed. No retries permitted until 2026-11-02 06:44:43.709989273 +0000 UTC m=+1660.158366018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-maas-api-tls") pod "maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" (UID: "6602a158-b770-4ce3-80d2-6c050ad2c591") : secret "maas-api-e2e-api-a-70898f-serving-cert" not found Nov 02 06:44:43.241223 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.237901 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:43.251141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.248511 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-metrics-certs\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.302874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.302836 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbxp\" (UniqueName: \"kubernetes.io/projected/6602a158-b770-4ce3-80d2-6c050ad2c591-kube-api-access-mlbxp\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.440224 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.440113 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:43.619697 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.619616 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:43.735090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.735042 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-maas-api-tls\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.737510 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.737490 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-maas-api-tls\") pod \"maas-api-e2e-api-a-70898f-d9476cfcf-8z25k\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:43.918120 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:43.918062 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:44.100101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:44.097526 5025 patch_prober.go:29] interesting pod/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:44:44.100101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:44.097606 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 06:44:44.312052 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:44:44.312014 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6602a158_b770_4ce3_80d2_6c050ad2c591.slice/crio-9ea800a6f0e176b46c67e186e5a1bb639d7b7e414d75f89aa2662babf25dff23 WatchSource:0}: Error finding container 9ea800a6f0e176b46c67e186e5a1bb639d7b7e414d75f89aa2662babf25dff23: Status 404 returned error can't find the container with id 9ea800a6f0e176b46c67e186e5a1bb639d7b7e414d75f89aa2662babf25dff23 Nov 02 06:44:44.312299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:44.312273 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k"] Nov 02 06:44:44.313592 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:44.313573 5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:44:44.413278 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:44.413232 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" event={"ID":"6602a158-b770-4ce3-80d2-6c050ad2c591","Type":"ContainerStarted","Data":"9ea800a6f0e176b46c67e186e5a1bb639d7b7e414d75f89aa2662babf25dff23"} Nov 02 06:44:45.108099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:45.107520 5025 patch_prober.go:29] interesting pod/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:44:45.108099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:45.107582 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 06:44:45.225012 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:45.224975 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:46.093134 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.090490 5025 patch_prober.go:29] interesting pod/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Nov 02 06:44:46.093134 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.090554 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Nov 02 06:44:46.160488 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.156439 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7123-b427a2"] Nov 02 06:44:46.190246 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.163853 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-b427a2" Nov 02 06:44:46.193397 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.193350 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-b427a2"] Nov 02 06:44:46.325419 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.296145 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrt72\" (UniqueName: \"kubernetes.io/projected/add1d1ef-9601-42f1-bade-093ffab41fbc-kube-api-access-mrt72\") pod \"test-curl-7123-b427a2\" (UID: \"add1d1ef-9601-42f1-bade-093ffab41fbc\") " pod="opendatahub/test-curl-7123-b427a2" Nov 02 06:44:46.325419 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.296770 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7g9gq\"" Nov 02 06:44:46.325419 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.297042 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 06:44:46.325419 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.297097 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 06:44:46.409151 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.408345 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrt72\" (UniqueName: \"kubernetes.io/projected/add1d1ef-9601-42f1-bade-093ffab41fbc-kube-api-access-mrt72\") pod \"test-curl-7123-b427a2\" (UID: \"add1d1ef-9601-42f1-bade-093ffab41fbc\") " pod="opendatahub/test-curl-7123-b427a2" Nov 02 06:44:46.432441 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.432395 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" event={"ID":"6602a158-b770-4ce3-80d2-6c050ad2c591","Type":"ContainerStarted","Data":"add4547793030e3b18adb9b109c6626c85150a0f5e9f2abc840ed140903b30f6"} Nov 02 06:44:46.432643 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.432547 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:46.464982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.464950 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrt72\" (UniqueName: \"kubernetes.io/projected/add1d1ef-9601-42f1-bade-093ffab41fbc-kube-api-access-mrt72\") pod \"test-curl-7123-b427a2\" (UID: \"add1d1ef-9601-42f1-bade-093ffab41fbc\") " pod="opendatahub/test-curl-7123-b427a2" Nov 02 06:44:46.548642 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.548556 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" podStartSLOduration=3.484187148 podStartE2EDuration="4.548541021s" totalImagesPullingTime="1.064353873s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:44:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:44:46.544521133 +0000 UTC m=+1662.992897889" watchObservedRunningTime="2026-11-02 06:44:46.548541021 +0000 UTC m=+1662.996917775" Nov 02 06:44:46.660149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:46.659371 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-b427a2" Nov 02 06:44:47.088254 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:47.088223 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:44:47.088732 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:47.088718 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:44:47.089561 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:47.089538 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:44:47.141560 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:44:47.141529 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadd1d1ef_9601_42f1_bade_093ffab41fbc.slice/crio-5d2398fdc64566ac849165e86648df405fb8b205e6edc9d75bb05bb62e49018c WatchSource:0}: Error finding container 5d2398fdc64566ac849165e86648df405fb8b205e6edc9d75bb05bb62e49018c: Status 404 returned error can't find the container with id 5d2398fdc64566ac849165e86648df405fb8b205e6edc9d75bb05bb62e49018c Nov 02 06:44:47.141695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:47.141568 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-b427a2"] Nov 02 06:44:47.469977 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:47.469917 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-b427a2" event={"ID":"add1d1ef-9601-42f1-bade-093ffab41fbc","Type":"ContainerStarted","Data":"014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367"} Nov 02 06:44:47.469977 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:47.469969 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-b427a2" event={"ID":"add1d1ef-9601-42f1-bade-093ffab41fbc","Type":"ContainerStarted","Data":"5d2398fdc64566ac849165e86648df405fb8b205e6edc9d75bb05bb62e49018c"} Nov 02 06:44:47.580409 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:47.580355 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7123-b427a2" podStartSLOduration=1.580341263 podStartE2EDuration="1.580341263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:44:47.578394683 +0000 UTC m=+1664.026771458" watchObservedRunningTime="2026-11-02 06:44:47.580341263 +0000 UTC m=+1664.028718016" Nov 02 06:44:48.366100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:48.365983 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:48.935842 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:48.935802 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7123-b427a2"] Nov 02 06:44:48.942182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:48.942150 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7123-b427a2"] Nov 02 06:44:49.107827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:49.107791 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7123-04591b"] Nov 02 06:44:49.108488 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:49.108472 5025 cpu_manager.go:414] "containerMap: removing container" podUID="add1d1ef-9601-42f1-bade-093ffab41fbc" containerName="test-curl-7123-b427a2" Nov 02 06:44:49.108551 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:49.108488 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="add1d1ef-9601-42f1-bade-093ffab41fbc" containerName="test-curl-7123-b427a2" Nov 02 06:44:49.108656 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:49.108646 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="add1d1ef-9601-42f1-bade-093ffab41fbc" containerName="test-curl-7123-b427a2" Nov 02 06:44:49.111338 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:49.111321 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:49.170465 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:49.170440 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-04591b"] Nov 02 06:44:49.280761 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:49.280673 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8z9w\" (UniqueName: \"kubernetes.io/projected/68b2a74d-7a99-4b99-a737-da200f5d798f-kube-api-access-s8z9w\") pod \"test-curl-7123-04591b\" (UID: \"68b2a74d-7a99-4b99-a737-da200f5d798f\") " pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:49.382695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:49.382663 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8z9w\" (UniqueName: \"kubernetes.io/projected/68b2a74d-7a99-4b99-a737-da200f5d798f-kube-api-access-s8z9w\") pod \"test-curl-7123-04591b\" (UID: \"68b2a74d-7a99-4b99-a737-da200f5d798f\") " pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:49.438627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:49.438593 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8z9w\" (UniqueName: \"kubernetes.io/projected/68b2a74d-7a99-4b99-a737-da200f5d798f-kube-api-access-s8z9w\") pod \"test-curl-7123-04591b\" (UID: \"68b2a74d-7a99-4b99-a737-da200f5d798f\") " pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:49.484794 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:49.484747 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7123-b427a2" podUID="add1d1ef-9601-42f1-bade-093ffab41fbc" containerName="test-curl-7123-b427a2" containerID="cri-o://014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367" gracePeriod=2 Nov 02 06:44:49.533414 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:49.533338 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-b427a2\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="add1d1ef-9601-42f1-bade-093ffab41fbc" pod="opendatahub/test-curl-7123-b427a2" Nov 02 06:44:49.722578 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:49.722548 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:50.091840 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:50.091811 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-04591b"] Nov 02 06:44:50.092263 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:44:50.092241 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68b2a74d_7a99_4b99_a737_da200f5d798f.slice/crio-5f4d04c09face52c59df0ae9f67fffa567e4854ac349295a0898ef72e161fd2e WatchSource:0}: Error finding container 5f4d04c09face52c59df0ae9f67fffa567e4854ac349295a0898ef72e161fd2e: Status 404 returned error can't find the container with id 5f4d04c09face52c59df0ae9f67fffa567e4854ac349295a0898ef72e161fd2e Nov 02 06:44:50.495937 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:50.495888 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-04591b" event={"ID":"68b2a74d-7a99-4b99-a737-da200f5d798f","Type":"ContainerStarted","Data":"3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed"} Nov 02 06:44:50.495937 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:50.495938 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-04591b" event={"ID":"68b2a74d-7a99-4b99-a737-da200f5d798f","Type":"ContainerStarted","Data":"5f4d04c09face52c59df0ae9f67fffa567e4854ac349295a0898ef72e161fd2e"} Nov 02 06:44:50.606021 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:50.605971 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7123-04591b" podStartSLOduration=1.6059579 podStartE2EDuration="1.6059579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:44:50.603504694 +0000 UTC m=+1667.051881468" watchObservedRunningTime="2026-11-02 06:44:50.6059579 +0000 UTC m=+1667.054334655" Nov 02 06:44:51.731828 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:51.731802 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-b427a2" Nov 02 06:44:51.923520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:51.923493 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/add1d1ef-9601-42f1-bade-093ffab41fbc-kube-api-access-mrt72\" (UniqueName: \"kubernetes.io/projected/add1d1ef-9601-42f1-bade-093ffab41fbc-kube-api-access-mrt72\") pod \"add1d1ef-9601-42f1-bade-093ffab41fbc\" (UID: \"add1d1ef-9601-42f1-bade-093ffab41fbc\") " Nov 02 06:44:51.925499 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:51.925477 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add1d1ef-9601-42f1-bade-093ffab41fbc-kube-api-access-mrt72" pod "add1d1ef-9601-42f1-bade-093ffab41fbc" (UID: "add1d1ef-9601-42f1-bade-093ffab41fbc"). InnerVolumeSpecName "kube-api-access-mrt72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:44:52.025165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.025124 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrt72\" (UniqueName: \"kubernetes.io/projected/add1d1ef-9601-42f1-bade-093ffab41fbc-kube-api-access-mrt72\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:52.031133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.031100 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7123-04591b"] Nov 02 06:44:52.035027 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.034997 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7123-04591b"] Nov 02 06:44:52.185305 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.185230 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7123-ed5198"] Nov 02 06:44:52.185957 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.185940 5025 cpu_manager.go:414] "containerMap: removing container" podUID="68b2a74d-7a99-4b99-a737-da200f5d798f" containerName="test-curl-7123-04591b" Nov 02 06:44:52.185957 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.185956 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b2a74d-7a99-4b99-a737-da200f5d798f" containerName="test-curl-7123-04591b" Nov 02 06:44:52.186133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.186122 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b2a74d-7a99-4b99-a737-da200f5d798f" containerName="test-curl-7123-04591b" Nov 02 06:44:52.189327 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.189308 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-ed5198" Nov 02 06:44:52.246474 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.246452 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-ed5198"] Nov 02 06:44:52.329852 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.329807 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fgn\" (UniqueName: \"kubernetes.io/projected/f85280db-95c8-4c95-ba3a-aea9ea03401e-kube-api-access-q7fgn\") pod \"test-curl-7123-ed5198\" (UID: \"f85280db-95c8-4c95-ba3a-aea9ea03401e\") " pod="opendatahub/test-curl-7123-ed5198" Nov 02 06:44:52.431226 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.431192 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fgn\" (UniqueName: \"kubernetes.io/projected/f85280db-95c8-4c95-ba3a-aea9ea03401e-kube-api-access-q7fgn\") pod \"test-curl-7123-ed5198\" (UID: \"f85280db-95c8-4c95-ba3a-aea9ea03401e\") " pod="opendatahub/test-curl-7123-ed5198" Nov 02 06:44:52.475875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.475806 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:44:52.485832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.485797 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fgn\" (UniqueName: \"kubernetes.io/projected/f85280db-95c8-4c95-ba3a-aea9ea03401e-kube-api-access-q7fgn\") pod \"test-curl-7123-ed5198\" (UID: \"f85280db-95c8-4c95-ba3a-aea9ea03401e\") " pod="opendatahub/test-curl-7123-ed5198" Nov 02 06:44:52.501294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.501267 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-ed5198" Nov 02 06:44:52.522390 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.522362 5025 generic.go:356] "Generic (PLEG): container finished" podID="add1d1ef-9601-42f1-bade-093ffab41fbc" containerID="014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367" exitCode=137 Nov 02 06:44:52.522491 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.522404 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-b427a2" Nov 02 06:44:52.522491 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.522457 5025 scope.go:122] "RemoveContainer" containerID="014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367" Nov 02 06:44:52.522704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.522660 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7123-04591b" podUID="68b2a74d-7a99-4b99-a737-da200f5d798f" containerName="test-curl-7123-04591b" containerID="cri-o://3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed" gracePeriod=2 Nov 02 06:44:52.548269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.548248 5025 scope.go:122] "RemoveContainer" containerID="014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367" Nov 02 06:44:52.548577 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:52.548549 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367\": container with ID starting with 014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367 not found: ID does not exist" containerID="014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367" Nov 02 06:44:52.548674 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.548588 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367"} err="failed to get container status \"014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367\": rpc error: code = NotFound desc = could not find container \"014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367\": container with ID starting with 014a9d425bb9ca9368bbef81aa5521966a7a5abf97fb30c4e8615e6f221d6367 not found: ID does not exist" Nov 02 06:44:52.638443 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:52.638367 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-04591b\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="68b2a74d-7a99-4b99-a737-da200f5d798f" pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:52.686684 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:52.686649 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-b427a2\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="add1d1ef-9601-42f1-bade-093ffab41fbc" pod="opendatahub/test-curl-7123-b427a2" Nov 02 06:44:52.879721 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.879683 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-ed5198"] Nov 02 06:44:52.880158 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:44:52.880134 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf85280db_95c8_4c95_ba3a_aea9ea03401e.slice/crio-cbf1f9cc24176cf736da405bb482484d45c35c22c0bab8d95545d2893910800e WatchSource:0}: Error finding container cbf1f9cc24176cf736da405bb482484d45c35c22c0bab8d95545d2893910800e: Status 404 returned error can't find the container with id cbf1f9cc24176cf736da405bb482484d45c35c22c0bab8d95545d2893910800e Nov 02 06:44:52.930109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:52.929623 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:53.531180 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:53.531130 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-ed5198" event={"ID":"f85280db-95c8-4c95-ba3a-aea9ea03401e","Type":"ContainerStarted","Data":"65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0"} Nov 02 06:44:53.531180 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:53.531179 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-ed5198" event={"ID":"f85280db-95c8-4c95-ba3a-aea9ea03401e","Type":"ContainerStarted","Data":"cbf1f9cc24176cf736da405bb482484d45c35c22c0bab8d95545d2893910800e"} Nov 02 06:44:53.579631 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:53.579589 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-b427a2\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="add1d1ef-9601-42f1-bade-093ffab41fbc" pod="opendatahub/test-curl-7123-b427a2" Nov 02 06:44:53.671117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:53.671054 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="add1d1ef-9601-42f1-bade-093ffab41fbc" path="/var/lib/kubelet/pods/add1d1ef-9601-42f1-bade-093ffab41fbc/volumes" Nov 02 06:44:53.690427 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:53.690375 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7123-ed5198" podStartSLOduration=1.690361743 podStartE2EDuration="1.690361743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:44:53.68766745 +0000 UTC m=+1670.136044221" watchObservedRunningTime="2026-11-02 06:44:53.690361743 +0000 UTC m=+1670.138738497" Nov 02 06:44:53.735897 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:53.735865 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-04591b\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="68b2a74d-7a99-4b99-a737-da200f5d798f" pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:54.776317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:54.776249 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:54.824733 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:54.824680 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-04591b\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="68b2a74d-7a99-4b99-a737-da200f5d798f" pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:54.870668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:54.870640 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b2a74d-7a99-4b99-a737-da200f5d798f-kube-api-access-s8z9w\" (UniqueName: \"kubernetes.io/projected/68b2a74d-7a99-4b99-a737-da200f5d798f-kube-api-access-s8z9w\") pod \"68b2a74d-7a99-4b99-a737-da200f5d798f\" (UID: \"68b2a74d-7a99-4b99-a737-da200f5d798f\") " Nov 02 06:44:54.872789 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:54.872763 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b2a74d-7a99-4b99-a737-da200f5d798f-kube-api-access-s8z9w" pod "68b2a74d-7a99-4b99-a737-da200f5d798f" (UID: "68b2a74d-7a99-4b99-a737-da200f5d798f"). InnerVolumeSpecName "kube-api-access-s8z9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:44:54.972168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:54.972140 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8z9w\" (UniqueName: \"kubernetes.io/projected/68b2a74d-7a99-4b99-a737-da200f5d798f-kube-api-access-s8z9w\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:55.003398 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.003375 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7123-ed5198"] Nov 02 06:44:55.008806 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.008779 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7123-ed5198"] Nov 02 06:44:55.155608 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.155531 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7123-1570fb"] Nov 02 06:44:55.156495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.156474 5025 cpu_manager.go:414] "containerMap: removing container" podUID="f85280db-95c8-4c95-ba3a-aea9ea03401e" containerName="test-curl-7123-ed5198" Nov 02 06:44:55.156495 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.156494 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f85280db-95c8-4c95-ba3a-aea9ea03401e" containerName="test-curl-7123-ed5198" Nov 02 06:44:55.156667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.156628 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="f85280db-95c8-4c95-ba3a-aea9ea03401e" containerName="test-curl-7123-ed5198" Nov 02 06:44:55.160230 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.160216 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-1570fb" Nov 02 06:44:55.174614 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.174592 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q62qx\" (UniqueName: \"kubernetes.io/projected/0696b57d-fd2f-4b0c-bf50-cefeaa51d99c-kube-api-access-q62qx\") pod \"test-curl-7123-1570fb\" (UID: \"0696b57d-fd2f-4b0c-bf50-cefeaa51d99c\") " pod="opendatahub/test-curl-7123-1570fb" Nov 02 06:44:55.208558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.208535 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-1570fb"] Nov 02 06:44:55.276645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.276597 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q62qx\" (UniqueName: \"kubernetes.io/projected/0696b57d-fd2f-4b0c-bf50-cefeaa51d99c-kube-api-access-q62qx\") pod \"test-curl-7123-1570fb\" (UID: \"0696b57d-fd2f-4b0c-bf50-cefeaa51d99c\") " pod="opendatahub/test-curl-7123-1570fb" Nov 02 06:44:55.313771 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:55.313735 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-04591b\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="68b2a74d-7a99-4b99-a737-da200f5d798f" pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:55.331465 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.331442 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q62qx\" (UniqueName: \"kubernetes.io/projected/0696b57d-fd2f-4b0c-bf50-cefeaa51d99c-kube-api-access-q62qx\") pod \"test-curl-7123-1570fb\" (UID: \"0696b57d-fd2f-4b0c-bf50-cefeaa51d99c\") " pod="opendatahub/test-curl-7123-1570fb" Nov 02 06:44:55.473149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.473112 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-1570fb" Nov 02 06:44:55.558714 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.558676 5025 generic.go:356] "Generic (PLEG): container finished" podID="68b2a74d-7a99-4b99-a737-da200f5d798f" containerID="3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed" exitCode=137 Nov 02 06:44:55.558949 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.558925 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7123-ed5198" podUID="f85280db-95c8-4c95-ba3a-aea9ea03401e" containerName="test-curl-7123-ed5198" containerID="cri-o://65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0" gracePeriod=2 Nov 02 06:44:55.559329 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.559308 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:55.562225 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.562198 5025 scope.go:122] "RemoveContainer" containerID="3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed" Nov 02 06:44:55.574251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.574233 5025 scope.go:122] "RemoveContainer" containerID="3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed" Nov 02 06:44:55.574587 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:55.574563 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed\": container with ID starting with 3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed not found: ID does not exist" containerID="3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed" Nov 02 06:44:55.574705 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.574592 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed"} err="failed to get container status \"3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed\": rpc error: code = NotFound desc = could not find container \"3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed\": container with ID starting with 3f614a4459193d1e2cdfdfb0527be69a6f0f7eee8d6245d0fb1857ca50ae7fed not found: ID does not exist" Nov 02 06:44:55.611817 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:55.611773 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-ed5198\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="f85280db-95c8-4c95-ba3a-aea9ea03401e" pod="opendatahub/test-curl-7123-ed5198" Nov 02 06:44:55.659870 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:55.659831 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-04591b\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="68b2a74d-7a99-4b99-a737-da200f5d798f" pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:55.671507 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.671478 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b2a74d-7a99-4b99-a737-da200f5d798f" path="/var/lib/kubelet/pods/68b2a74d-7a99-4b99-a737-da200f5d798f/volumes" Nov 02 06:44:55.708762 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:55.708721 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-ed5198\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="f85280db-95c8-4c95-ba3a-aea9ea03401e" pod="opendatahub/test-curl-7123-ed5198" Nov 02 06:44:55.757109 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:55.757025 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-04591b\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="68b2a74d-7a99-4b99-a737-da200f5d798f" pod="opendatahub/test-curl-7123-04591b" Nov 02 06:44:55.840661 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:55.840632 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7123-1570fb"] Nov 02 06:44:55.841171 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:44:55.841147 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0696b57d_fd2f_4b0c_bf50_cefeaa51d99c.slice/crio-9974fbd3eabfd0d48ffd4e9c5ed6f55551822ad4713c519811cb0c034560dde7 WatchSource:0}: Error finding container 9974fbd3eabfd0d48ffd4e9c5ed6f55551822ad4713c519811cb0c034560dde7: Status 404 returned error can't find the container with id 9974fbd3eabfd0d48ffd4e9c5ed6f55551822ad4713c519811cb0c034560dde7 Nov 02 06:44:56.568833 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:56.568792 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-1570fb" event={"ID":"0696b57d-fd2f-4b0c-bf50-cefeaa51d99c","Type":"ContainerStarted","Data":"f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5"} Nov 02 06:44:56.569003 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:56.568843 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7123-1570fb" event={"ID":"0696b57d-fd2f-4b0c-bf50-cefeaa51d99c","Type":"ContainerStarted","Data":"9974fbd3eabfd0d48ffd4e9c5ed6f55551822ad4713c519811cb0c034560dde7"} Nov 02 06:44:56.676654 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:56.676602 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7123-1570fb" podStartSLOduration=1.67658895 podStartE2EDuration="1.67658895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:44:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:44:56.675733407 +0000 UTC m=+1673.124110163" watchObservedRunningTime="2026-11-02 06:44:56.67658895 +0000 UTC m=+1673.124965700" Nov 02 06:44:57.531101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:57.526712 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:57.808818 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:57.808793 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-ed5198" Nov 02 06:44:57.919578 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:57.919549 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f85280db-95c8-4c95-ba3a-aea9ea03401e-kube-api-access-q7fgn\" (UniqueName: \"kubernetes.io/projected/f85280db-95c8-4c95-ba3a-aea9ea03401e-kube-api-access-q7fgn\") pod \"f85280db-95c8-4c95-ba3a-aea9ea03401e\" (UID: \"f85280db-95c8-4c95-ba3a-aea9ea03401e\") " Nov 02 06:44:57.921507 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:57.921481 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f85280db-95c8-4c95-ba3a-aea9ea03401e-kube-api-access-q7fgn" pod "f85280db-95c8-4c95-ba3a-aea9ea03401e" (UID: "f85280db-95c8-4c95-ba3a-aea9ea03401e"). InnerVolumeSpecName "kube-api-access-q7fgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:44:58.021805 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:58.021771 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7fgn\" (UniqueName: \"kubernetes.io/projected/f85280db-95c8-4c95-ba3a-aea9ea03401e-kube-api-access-q7fgn\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:44:58.101440 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:58.101362 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7123-1570fb"] Nov 02 06:44:58.105208 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:58.105176 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7123-1570fb"] Nov 02 06:44:58.597205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:58.597158 5025 generic.go:356] "Generic (PLEG): container finished" podID="f85280db-95c8-4c95-ba3a-aea9ea03401e" containerID="65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0" exitCode=137 Nov 02 06:44:58.597205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:58.597194 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-ed5198" Nov 02 06:44:58.597741 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:58.597246 5025 scope.go:122] "RemoveContainer" containerID="65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0" Nov 02 06:44:58.597741 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:58.597454 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7123-1570fb" podUID="0696b57d-fd2f-4b0c-bf50-cefeaa51d99c" containerName="test-curl-7123-1570fb" containerID="cri-o://f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5" gracePeriod=2 Nov 02 06:44:58.608659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:58.608633 5025 scope.go:122] "RemoveContainer" containerID="65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0" Nov 02 06:44:58.608923 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:58.608904 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0\": container with ID starting with 65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0 not found: ID does not exist" containerID="65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0" Nov 02 06:44:58.608995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:58.608931 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0"} err="failed to get container status \"65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0\": rpc error: code = NotFound desc = could not find container \"65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0\": container with ID starting with 65b5c3ae351a086fa92715eeb2649f97bd745d1dded98e8371d629cae57876c0 not found: ID does not exist" Nov 02 06:44:58.645875 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:58.645848 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-1570fb\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="0696b57d-fd2f-4b0c-bf50-cefeaa51d99c" pod="opendatahub/test-curl-7123-1570fb" Nov 02 06:44:58.664712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:58.664688 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:58.694229 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:58.694199 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-1570fb\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="0696b57d-fd2f-4b0c-bf50-cefeaa51d99c" pod="opendatahub/test-curl-7123-1570fb" Nov 02 06:44:58.742847 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:58.742813 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-ed5198\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="f85280db-95c8-4c95-ba3a-aea9ea03401e" pod="opendatahub/test-curl-7123-ed5198" Nov 02 06:44:59.129197 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.129160 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg"] Nov 02 06:44:59.130011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.129989 5025 cpu_manager.go:414] "containerMap: removing container" podUID="0696b57d-fd2f-4b0c-bf50-cefeaa51d99c" containerName="test-curl-7123-1570fb" Nov 02 06:44:59.130011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.130011 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0696b57d-fd2f-4b0c-bf50-cefeaa51d99c" containerName="test-curl-7123-1570fb" Nov 02 06:44:59.130234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.130187 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="0696b57d-fd2f-4b0c-bf50-cefeaa51d99c" containerName="test-curl-7123-1570fb" Nov 02 06:44:59.134990 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.134971 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.140544 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.140517 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zrpz\" (UniqueName: \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-kube-api-access-6zrpz\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.140662 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.140574 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-data\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.140708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.140659 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-envoy\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.140852 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.140832 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-podinfo\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.140904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.140890 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-token\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.140943 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.140924 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-credential-socket\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.141061 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.141044 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-certs\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.141143 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.141130 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-socket\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.141177 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.141157 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0e3851-c4c8-4bed-971c-97f54574135d-istiod-ca-cert\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.194098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.193441 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-12bfb5-openshift-default-dockercfg-ltg9x\"" Nov 02 06:44:59.194098 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:59.193552 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-1570fb\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="0696b57d-fd2f-4b0c-bf50-cefeaa51d99c" pod="opendatahub/test-curl-7123-1570fb" Nov 02 06:44:59.321205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.304741 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg"] Nov 02 06:44:59.342121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.341535 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.398927 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-socket\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.398973 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0e3851-c4c8-4bed-971c-97f54574135d-istiod-ca-cert\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.399055 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zrpz\" (UniqueName: \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-kube-api-access-6zrpz\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.399132 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-data\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.399175 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-envoy\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.399328 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-podinfo\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.399359 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-socket\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.399380 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-token\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.399411 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-credential-socket\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.399500 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-certs\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.399611 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-data\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.401553 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-credential-socket\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.401901 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-certs\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.407968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.402344 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0e3851-c4c8-4bed-971c-97f54574135d-istiod-ca-cert\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.432807 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:44:59.418834 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7123-ed5198\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="f85280db-95c8-4c95-ba3a-aea9ea03401e" pod="opendatahub/test-curl-7123-ed5198" Nov 02 06:44:59.518110 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.475479 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-envoy\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.565241 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.565135 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-podinfo\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.601175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.601092 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zrpz\" (UniqueName: \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-kube-api-access-6zrpz\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.626127 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.617602 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-token\") pod \"e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.669847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.669815 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f85280db-95c8-4c95-ba3a-aea9ea03401e" path="/var/lib/kubelet/pods/f85280db-95c8-4c95-ba3a-aea9ea03401e/volumes" Nov 02 06:44:59.750171 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.750138 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:44:59.827930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:44:59.826507 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:00.159540 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.159455 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb"] Nov 02 06:45:00.165380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.165358 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" Nov 02 06:45:00.189446 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.189423 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n"] Nov 02 06:45:00.194971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.194952 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" Nov 02 06:45:00.211097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.211062 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5"] Nov 02 06:45:00.215436 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.215417 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" Nov 02 06:45:00.220460 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.220442 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 06:45:00.230539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.230520 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh"] Nov 02 06:45:00.234531 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.234516 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" Nov 02 06:45:00.256535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.250632 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38020a87-e892-4241-8251-c1bff66bac2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb\" (UID: \"38020a87-e892-4241-8251-c1bff66bac2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" Nov 02 06:45:00.256535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.250911 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl5pt\" (UniqueName: \"kubernetes.io/projected/9129a2af-4c8c-4aba-a126-dc7af1199ddb-kube-api-access-jl5pt\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n\" (UID: \"9129a2af-4c8c-4aba-a126-dc7af1199ddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" Nov 02 06:45:00.256535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.250999 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9129a2af-4c8c-4aba-a126-dc7af1199ddb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n\" (UID: \"9129a2af-4c8c-4aba-a126-dc7af1199ddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" Nov 02 06:45:00.256535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.251203 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7954\" (UniqueName: \"kubernetes.io/projected/38020a87-e892-4241-8251-c1bff66bac2b-kube-api-access-t7954\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb\" (UID: \"38020a87-e892-4241-8251-c1bff66bac2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" Nov 02 06:45:00.256535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.252882 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb"] Nov 02 06:45:00.260498 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.260474 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n"] Nov 02 06:45:00.270933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.270908 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5"] Nov 02 06:45:00.284472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.284447 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg"] Nov 02 06:45:00.298265 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.298241 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh"] Nov 02 06:45:00.309003 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:45:00.308943 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0e3851_c4c8_4bed_971c_97f54574135d.slice/crio-6a3b9fe32a255c78cb63adb2446fb25f308cfd389a3517e58fa9e8724c951c29 WatchSource:0}: Error finding container 6a3b9fe32a255c78cb63adb2446fb25f308cfd389a3517e58fa9e8724c951c29: Status 404 returned error can't find the container with id 6a3b9fe32a255c78cb63adb2446fb25f308cfd389a3517e58fa9e8724c951c29 Nov 02 06:45:00.313525 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.313489 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:45:00.313641 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.313556 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:45:00.313641 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.313587 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:45:00.353259 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.353232 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh\" (UID: \"7aa1a3bf-856c-4b15-8c4c-1e522da49fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" Nov 02 06:45:00.353409 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.353300 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5pt\" (UniqueName: \"kubernetes.io/projected/9129a2af-4c8c-4aba-a126-dc7af1199ddb-kube-api-access-jl5pt\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n\" (UID: \"9129a2af-4c8c-4aba-a126-dc7af1199ddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" Nov 02 06:45:00.353409 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.353375 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9129a2af-4c8c-4aba-a126-dc7af1199ddb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n\" (UID: \"9129a2af-4c8c-4aba-a126-dc7af1199ddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" Nov 02 06:45:00.353529 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.353456 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5\" (UID: \"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" Nov 02 06:45:00.353587 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.353548 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7954\" (UniqueName: \"kubernetes.io/projected/38020a87-e892-4241-8251-c1bff66bac2b-kube-api-access-t7954\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb\" (UID: \"38020a87-e892-4241-8251-c1bff66bac2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" Nov 02 06:45:00.353734 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.353717 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc6jm\" (UniqueName: \"kubernetes.io/projected/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-kube-api-access-dc6jm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh\" (UID: \"7aa1a3bf-856c-4b15-8c4c-1e522da49fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" Nov 02 06:45:00.353795 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.353780 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tczm\" (UniqueName: \"kubernetes.io/projected/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-kube-api-access-5tczm\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5\" (UID: \"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" Nov 02 06:45:00.353855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.353812 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38020a87-e892-4241-8251-c1bff66bac2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb\" (UID: \"38020a87-e892-4241-8251-c1bff66bac2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" Nov 02 06:45:00.354236 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.354210 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9129a2af-4c8c-4aba-a126-dc7af1199ddb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n\" (UID: \"9129a2af-4c8c-4aba-a126-dc7af1199ddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" Nov 02 06:45:00.354641 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.354614 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38020a87-e892-4241-8251-c1bff66bac2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb\" (UID: \"38020a87-e892-4241-8251-c1bff66bac2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" Nov 02 06:45:00.407556 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.407527 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl5pt\" (UniqueName: \"kubernetes.io/projected/9129a2af-4c8c-4aba-a126-dc7af1199ddb-kube-api-access-jl5pt\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n\" (UID: \"9129a2af-4c8c-4aba-a126-dc7af1199ddb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" Nov 02 06:45:00.407764 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.407747 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7954\" (UniqueName: \"kubernetes.io/projected/38020a87-e892-4241-8251-c1bff66bac2b-kube-api-access-t7954\") pod \"maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb\" (UID: \"38020a87-e892-4241-8251-c1bff66bac2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" Nov 02 06:45:00.455985 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.455951 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5\" (UID: \"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" Nov 02 06:45:00.456219 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.456182 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc6jm\" (UniqueName: \"kubernetes.io/projected/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-kube-api-access-dc6jm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh\" (UID: \"7aa1a3bf-856c-4b15-8c4c-1e522da49fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" Nov 02 06:45:00.456297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.456230 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tczm\" (UniqueName: \"kubernetes.io/projected/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-kube-api-access-5tczm\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5\" (UID: \"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" Nov 02 06:45:00.456354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.456320 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh\" (UID: \"7aa1a3bf-856c-4b15-8c4c-1e522da49fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" Nov 02 06:45:00.456712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.456687 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5\" (UID: \"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" Nov 02 06:45:00.457124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.457105 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh\" (UID: \"7aa1a3bf-856c-4b15-8c4c-1e522da49fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" Nov 02 06:45:00.483592 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.483567 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" Nov 02 06:45:00.509553 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.509520 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" Nov 02 06:45:00.513378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.513305 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tczm\" (UniqueName: \"kubernetes.io/projected/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-kube-api-access-5tczm\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5\" (UID: \"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" Nov 02 06:45:00.513938 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.513916 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc6jm\" (UniqueName: \"kubernetes.io/projected/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-kube-api-access-dc6jm\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh\" (UID: \"7aa1a3bf-856c-4b15-8c4c-1e522da49fff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" Nov 02 06:45:00.528950 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.528920 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" Nov 02 06:45:00.552524 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.552463 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" Nov 02 06:45:00.665399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.665355 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" event={"ID":"1c0e3851-c4c8-4bed-971c-97f54574135d","Type":"ContainerStarted","Data":"976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4"} Nov 02 06:45:00.665836 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.665422 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" event={"ID":"1c0e3851-c4c8-4bed-971c-97f54574135d","Type":"ContainerStarted","Data":"6a3b9fe32a255c78cb63adb2446fb25f308cfd389a3517e58fa9e8724c951c29"} Nov 02 06:45:00.750321 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.750274 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:45:00.787355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.787293 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" podStartSLOduration=1.7872743679999998 podStartE2EDuration="1.787274368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:00.781207625 +0000 UTC m=+1677.229584382" watchObservedRunningTime="2026-11-02 06:45:00.787274368 +0000 UTC m=+1677.235651125" Nov 02 06:45:00.868864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:00.868633 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb"] Nov 02 06:45:00.872369 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:45:00.872342 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38020a87_e892_4241_8251_c1bff66bac2b.slice/crio-ac423f1eb675342ce927cd26d4c10b6662642566db50e5242e96d666f9f98665 WatchSource:0}: Error finding container ac423f1eb675342ce927cd26d4c10b6662642566db50e5242e96d666f9f98665: Status 404 returned error can't find the container with id ac423f1eb675342ce927cd26d4c10b6662642566db50e5242e96d666f9f98665 Nov 02 06:45:01.001106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.001044 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n"] Nov 02 06:45:01.022959 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.022777 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-1570fb" Nov 02 06:45:01.071871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.071843 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh"] Nov 02 06:45:01.073228 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:45:01.073206 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa1a3bf_856c_4b15_8c4c_1e522da49fff.slice/crio-6742e77a3705a2bf98d317c21dcd87a0062acd1d55166e9cc5b08d198121c68b WatchSource:0}: Error finding container 6742e77a3705a2bf98d317c21dcd87a0062acd1d55166e9cc5b08d198121c68b: Status 404 returned error can't find the container with id 6742e77a3705a2bf98d317c21dcd87a0062acd1d55166e9cc5b08d198121c68b Nov 02 06:45:01.100918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.100825 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0696b57d-fd2f-4b0c-bf50-cefeaa51d99c-kube-api-access-q62qx\" (UniqueName: \"kubernetes.io/projected/0696b57d-fd2f-4b0c-bf50-cefeaa51d99c-kube-api-access-q62qx\") pod \"0696b57d-fd2f-4b0c-bf50-cefeaa51d99c\" (UID: \"0696b57d-fd2f-4b0c-bf50-cefeaa51d99c\") " Nov 02 06:45:01.117633 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.117608 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5"] Nov 02 06:45:01.185860 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.185825 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0696b57d-fd2f-4b0c-bf50-cefeaa51d99c-kube-api-access-q62qx" pod "0696b57d-fd2f-4b0c-bf50-cefeaa51d99c" (UID: "0696b57d-fd2f-4b0c-bf50-cefeaa51d99c"). InnerVolumeSpecName "kube-api-access-q62qx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:01.187971 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:45:01.187946 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f8a1f2_caa0_4b2a_be5f_c92187dfdbbb.slice/crio-4f282df32e5335c52e162031a95b1e52ffa86ce3afeac2f991e55ca28ab902fa WatchSource:0}: Error finding container 4f282df32e5335c52e162031a95b1e52ffa86ce3afeac2f991e55ca28ab902fa: Status 404 returned error can't find the container with id 4f282df32e5335c52e162031a95b1e52ffa86ce3afeac2f991e55ca28ab902fa Nov 02 06:45:01.202384 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.202362 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q62qx\" (UniqueName: \"kubernetes.io/projected/0696b57d-fd2f-4b0c-bf50-cefeaa51d99c-kube-api-access-q62qx\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:01.675599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.675549 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0696b57d-fd2f-4b0c-bf50-cefeaa51d99c" path="/var/lib/kubelet/pods/0696b57d-fd2f-4b0c-bf50-cefeaa51d99c/volumes" Nov 02 06:45:01.676867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.676838 5025 generic.go:356] "Generic (PLEG): container finished" podID="0696b57d-fd2f-4b0c-bf50-cefeaa51d99c" containerID="f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5" exitCode=137 Nov 02 06:45:01.677033 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.676890 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7123-1570fb" Nov 02 06:45:01.677033 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.676917 5025 scope.go:122] "RemoveContainer" containerID="f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5" Nov 02 06:45:01.678862 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.678830 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" event={"ID":"7aa1a3bf-856c-4b15-8c4c-1e522da49fff","Type":"ContainerStarted","Data":"7327c622ce16e513290e87cb88baa359073c008512adfe5229bcf9167f6a7963"} Nov 02 06:45:01.679002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.678891 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" event={"ID":"7aa1a3bf-856c-4b15-8c4c-1e522da49fff","Type":"ContainerStarted","Data":"6742e77a3705a2bf98d317c21dcd87a0062acd1d55166e9cc5b08d198121c68b"} Nov 02 06:45:01.680458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.680429 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" event={"ID":"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb","Type":"ContainerStarted","Data":"4f282df32e5335c52e162031a95b1e52ffa86ce3afeac2f991e55ca28ab902fa"} Nov 02 06:45:01.682475 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.682451 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" event={"ID":"9129a2af-4c8c-4aba-a126-dc7af1199ddb","Type":"ContainerStarted","Data":"9d395bb67c13daea4a40a80f76c35ca6301c3cb6472abd6eb79a594f37889ad1"} Nov 02 06:45:01.682591 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.682500 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" event={"ID":"9129a2af-4c8c-4aba-a126-dc7af1199ddb","Type":"ContainerStarted","Data":"cd88f674ec80990f4c2f1bad4592333d9d5450cc6b55893aea8a0627b2bf5540"} Nov 02 06:45:01.684664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.684562 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" event={"ID":"38020a87-e892-4241-8251-c1bff66bac2b","Type":"ContainerStarted","Data":"80c42b831ca4b2c95ba6158e9522162c6907e25d160d52beffbd0b91deac7e7a"} Nov 02 06:45:01.684664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.684618 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" event={"ID":"38020a87-e892-4241-8251-c1bff66bac2b","Type":"ContainerStarted","Data":"ac423f1eb675342ce927cd26d4c10b6662642566db50e5242e96d666f9f98665"} Nov 02 06:45:01.690576 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.690561 5025 scope.go:122] "RemoveContainer" containerID="f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5" Nov 02 06:45:01.690819 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:01.690800 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5\": container with ID starting with f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5 not found: ID does not exist" containerID="f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5" Nov 02 06:45:01.690879 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.690829 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5"} err="failed to get container status \"f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5\": rpc error: code = NotFound desc = could not find container \"f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5\": container with ID starting with f3340c9a52048694d0bd010125e04f86ae5ced6ed6f06c3a6f74d2637643daa5 not found: ID does not exist" Nov 02 06:45:01.756386 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.756353 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:45:01.789359 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.789306 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" podStartSLOduration=1.789293268 podStartE2EDuration="1.789293268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:01.786510174 +0000 UTC m=+1678.234886929" watchObservedRunningTime="2026-11-02 06:45:01.789293268 +0000 UTC m=+1678.237670023" Nov 02 06:45:01.895664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:01.895611 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" podStartSLOduration=1.895597253 podStartE2EDuration="1.895597253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:01.894614122 +0000 UTC m=+1678.342990879" watchObservedRunningTime="2026-11-02 06:45:01.895597253 +0000 UTC m=+1678.343974010" Nov 02 06:45:02.115341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.115293 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" podStartSLOduration=2.115278256 podStartE2EDuration="2.115278256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:02.114590408 +0000 UTC m=+1678.562967181" watchObservedRunningTime="2026-11-02 06:45:02.115278256 +0000 UTC m=+1678.563655012" Nov 02 06:45:02.315275 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.315139 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:02.693345 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.693305 5025 generic.go:356] "Generic (PLEG): container finished" podID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerID="7327c622ce16e513290e87cb88baa359073c008512adfe5229bcf9167f6a7963" exitCode=60 Nov 02 06:45:02.693852 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.693398 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" event={"ID":"7aa1a3bf-856c-4b15-8c4c-1e522da49fff","Type":"ContainerDied","Data":"7327c622ce16e513290e87cb88baa359073c008512adfe5229bcf9167f6a7963"} Nov 02 06:45:02.693852 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.693730 5025 scope.go:122] "RemoveContainer" containerID="7327c622ce16e513290e87cb88baa359073c008512adfe5229bcf9167f6a7963" Nov 02 06:45:02.695958 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.695931 5025 generic.go:356] "Generic (PLEG): container finished" podID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerID="f8b7a4de2d12f732b55c8cd7b2a42e0da83f0d0e5af145ad410c632ae5e329f5" exitCode=60 Nov 02 06:45:02.696509 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.696492 5025 scope.go:122] "RemoveContainer" containerID="f8b7a4de2d12f732b55c8cd7b2a42e0da83f0d0e5af145ad410c632ae5e329f5" Nov 02 06:45:02.696598 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.696531 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" event={"ID":"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb","Type":"ContainerDied","Data":"f8b7a4de2d12f732b55c8cd7b2a42e0da83f0d0e5af145ad410c632ae5e329f5"} Nov 02 06:45:02.700442 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.700416 5025 generic.go:356] "Generic (PLEG): container finished" podID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerID="9d395bb67c13daea4a40a80f76c35ca6301c3cb6472abd6eb79a594f37889ad1" exitCode=60 Nov 02 06:45:02.700605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.700502 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" event={"ID":"9129a2af-4c8c-4aba-a126-dc7af1199ddb","Type":"ContainerDied","Data":"9d395bb67c13daea4a40a80f76c35ca6301c3cb6472abd6eb79a594f37889ad1"} Nov 02 06:45:02.701235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.701001 5025 scope.go:122] "RemoveContainer" containerID="9d395bb67c13daea4a40a80f76c35ca6301c3cb6472abd6eb79a594f37889ad1" Nov 02 06:45:02.703738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.703715 5025 generic.go:356] "Generic (PLEG): container finished" podID="38020a87-e892-4241-8251-c1bff66bac2b" containerID="80c42b831ca4b2c95ba6158e9522162c6907e25d160d52beffbd0b91deac7e7a" exitCode=60 Nov 02 06:45:02.704028 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.704008 5025 scope.go:122] "RemoveContainer" containerID="80c42b831ca4b2c95ba6158e9522162c6907e25d160d52beffbd0b91deac7e7a" Nov 02 06:45:02.704614 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.703783 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" event={"ID":"38020a87-e892-4241-8251-c1bff66bac2b","Type":"ContainerDied","Data":"80c42b831ca4b2c95ba6158e9522162c6907e25d160d52beffbd0b91deac7e7a"} Nov 02 06:45:02.704710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.704674 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:45:02.705777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:02.705757 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:45:02.715135 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:02.715110 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:45:03.713027 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.712991 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" event={"ID":"7aa1a3bf-856c-4b15-8c4c-1e522da49fff","Type":"ContainerStarted","Data":"0f1413eee32f13ec3d1601f2b51d6c5a1c68a68d77e8f35c1f6648a20dda768c"} Nov 02 06:45:03.714997 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.714974 5025 generic.go:356] "Generic (PLEG): container finished" podID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerID="99c6083792697acdf544735fc66e52e975ef6c88da3b0e8555a72bf62789227f" exitCode=60 Nov 02 06:45:03.715132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.715053 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" event={"ID":"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb","Type":"ContainerDied","Data":"99c6083792697acdf544735fc66e52e975ef6c88da3b0e8555a72bf62789227f"} Nov 02 06:45:03.715202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.715139 5025 scope.go:122] "RemoveContainer" containerID="f8b7a4de2d12f732b55c8cd7b2a42e0da83f0d0e5af145ad410c632ae5e329f5" Nov 02 06:45:03.715362 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.715345 5025 scope.go:122] "RemoveContainer" containerID="99c6083792697acdf544735fc66e52e975ef6c88da3b0e8555a72bf62789227f" Nov 02 06:45:03.715750 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:03.715659 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5_odh-ai-gateway-infra(b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" Nov 02 06:45:03.717555 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.717536 5025 generic.go:356] "Generic (PLEG): container finished" podID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerID="9fa1f5b4cad9817cc99817386956f17691392fd04bfa86b432e972ca456e1d66" exitCode=60 Nov 02 06:45:03.717637 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.717555 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" event={"ID":"9129a2af-4c8c-4aba-a126-dc7af1199ddb","Type":"ContainerDied","Data":"9fa1f5b4cad9817cc99817386956f17691392fd04bfa86b432e972ca456e1d66"} Nov 02 06:45:03.717872 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.717854 5025 scope.go:122] "RemoveContainer" containerID="9fa1f5b4cad9817cc99817386956f17691392fd04bfa86b432e972ca456e1d66" Nov 02 06:45:03.718161 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:03.718139 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n_odh-ai-gateway-infra(9129a2af-4c8c-4aba-a126-dc7af1199ddb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" Nov 02 06:45:03.719577 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.719554 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" event={"ID":"38020a87-e892-4241-8251-c1bff66bac2b","Type":"ContainerStarted","Data":"035c71b934454315f9d8a0ff6776fcacb403f9aa7ff4759a2dc721f3acbd62b6"} Nov 02 06:45:03.761825 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.761798 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:03.898274 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.898251 5025 scope.go:122] "RemoveContainer" containerID="9d395bb67c13daea4a40a80f76c35ca6301c3cb6472abd6eb79a594f37889ad1" Nov 02 06:45:03.962115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:03.962069 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:04.728514 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:04.728476 5025 generic.go:356] "Generic (PLEG): container finished" podID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerID="0f1413eee32f13ec3d1601f2b51d6c5a1c68a68d77e8f35c1f6648a20dda768c" exitCode=60 Nov 02 06:45:04.728996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:04.728575 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" event={"ID":"7aa1a3bf-856c-4b15-8c4c-1e522da49fff","Type":"ContainerDied","Data":"0f1413eee32f13ec3d1601f2b51d6c5a1c68a68d77e8f35c1f6648a20dda768c"} Nov 02 06:45:04.728996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:04.728695 5025 scope.go:122] "RemoveContainer" containerID="7327c622ce16e513290e87cb88baa359073c008512adfe5229bcf9167f6a7963" Nov 02 06:45:04.728996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:04.728909 5025 scope.go:122] "RemoveContainer" containerID="0f1413eee32f13ec3d1601f2b51d6c5a1c68a68d77e8f35c1f6648a20dda768c" Nov 02 06:45:04.729560 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:04.729487 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh_odh-ai-gateway-infra(7aa1a3bf-856c-4b15-8c4c-1e522da49fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" Nov 02 06:45:04.736356 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:04.736322 5025 generic.go:356] "Generic (PLEG): container finished" podID="38020a87-e892-4241-8251-c1bff66bac2b" containerID="035c71b934454315f9d8a0ff6776fcacb403f9aa7ff4759a2dc721f3acbd62b6" exitCode=60 Nov 02 06:45:04.736940 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:04.736911 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" event={"ID":"38020a87-e892-4241-8251-c1bff66bac2b","Type":"ContainerDied","Data":"035c71b934454315f9d8a0ff6776fcacb403f9aa7ff4759a2dc721f3acbd62b6"} Nov 02 06:45:04.743780 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:04.743745 5025 scope.go:122] "RemoveContainer" containerID="99c6083792697acdf544735fc66e52e975ef6c88da3b0e8555a72bf62789227f" Nov 02 06:45:04.743979 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:04.743961 5025 scope.go:122] "RemoveContainer" containerID="9fa1f5b4cad9817cc99817386956f17691392fd04bfa86b432e972ca456e1d66" Nov 02 06:45:04.744202 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:04.744177 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5_odh-ai-gateway-infra(b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" Nov 02 06:45:04.744320 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:04.744306 5025 scope.go:122] "RemoveContainer" containerID="035c71b934454315f9d8a0ff6776fcacb403f9aa7ff4759a2dc721f3acbd62b6" Nov 02 06:45:04.744394 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:04.744306 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n_odh-ai-gateway-infra(9129a2af-4c8c-4aba-a126-dc7af1199ddb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" Nov 02 06:45:04.744532 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:04.744516 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb_odh-ai-gateway-infra(38020a87-e892-4241-8251-c1bff66bac2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" podUID="38020a87-e892-4241-8251-c1bff66bac2b" Nov 02 06:45:04.755706 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:04.755683 5025 scope.go:122] "RemoveContainer" containerID="80c42b831ca4b2c95ba6158e9522162c6907e25d160d52beffbd0b91deac7e7a" Nov 02 06:45:05.746171 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:05.746143 5025 scope.go:122] "RemoveContainer" containerID="035c71b934454315f9d8a0ff6776fcacb403f9aa7ff4759a2dc721f3acbd62b6" Nov 02 06:45:05.746598 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:05.746542 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb_odh-ai-gateway-infra(38020a87-e892-4241-8251-c1bff66bac2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" podUID="38020a87-e892-4241-8251-c1bff66bac2b" Nov 02 06:45:05.747801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:05.747779 5025 scope.go:122] "RemoveContainer" containerID="0f1413eee32f13ec3d1601f2b51d6c5a1c68a68d77e8f35c1f6648a20dda768c" Nov 02 06:45:05.748155 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:05.748135 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh_odh-ai-gateway-infra(7aa1a3bf-856c-4b15-8c4c-1e522da49fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" Nov 02 06:45:06.929708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:06.929675 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:07.109925 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:07.109889 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:07.505194 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:07.487993 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:12.169598 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:12.168544 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:12.351369 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:12.351340 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:12.617873 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:12.617800 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:15.662507 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:15.662471 5025 scope.go:122] "RemoveContainer" containerID="0f1413eee32f13ec3d1601f2b51d6c5a1c68a68d77e8f35c1f6648a20dda768c" Nov 02 06:45:16.662229 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:16.662195 5025 scope.go:122] "RemoveContainer" containerID="9fa1f5b4cad9817cc99817386956f17691392fd04bfa86b432e972ca456e1d66" Nov 02 06:45:16.884450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:16.884410 5025 generic.go:356] "Generic (PLEG): container finished" podID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerID="ef133022f6b2f1d5c08b12be520ed683409ed39e0d7f8d8cb76b7cbfb86c5f9e" exitCode=60 Nov 02 06:45:16.884950 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:16.884616 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" event={"ID":"7aa1a3bf-856c-4b15-8c4c-1e522da49fff","Type":"ContainerDied","Data":"ef133022f6b2f1d5c08b12be520ed683409ed39e0d7f8d8cb76b7cbfb86c5f9e"} Nov 02 06:45:16.884950 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:16.884678 5025 scope.go:122] "RemoveContainer" containerID="0f1413eee32f13ec3d1601f2b51d6c5a1c68a68d77e8f35c1f6648a20dda768c" Nov 02 06:45:16.887307 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:16.885338 5025 scope.go:122] "RemoveContainer" containerID="ef133022f6b2f1d5c08b12be520ed683409ed39e0d7f8d8cb76b7cbfb86c5f9e" Nov 02 06:45:16.887307 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:16.886153 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh_odh-ai-gateway-infra(7aa1a3bf-856c-4b15-8c4c-1e522da49fff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" Nov 02 06:45:17.163745 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.163705 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:17.671207 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.671180 5025 scope.go:122] "RemoveContainer" containerID="035c71b934454315f9d8a0ff6776fcacb403f9aa7ff4759a2dc721f3acbd62b6" Nov 02 06:45:17.894040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.893972 5025 generic.go:356] "Generic (PLEG): container finished" podID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerID="ce49dbc7f7cba4dbab637455ab8130719c4e42ef4cd0688b4012c4e0d7cd1a91" exitCode=60 Nov 02 06:45:17.894463 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.894041 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" event={"ID":"9129a2af-4c8c-4aba-a126-dc7af1199ddb","Type":"ContainerDied","Data":"ce49dbc7f7cba4dbab637455ab8130719c4e42ef4cd0688b4012c4e0d7cd1a91"} Nov 02 06:45:17.894463 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.894142 5025 scope.go:122] "RemoveContainer" containerID="9fa1f5b4cad9817cc99817386956f17691392fd04bfa86b432e972ca456e1d66" Nov 02 06:45:17.894582 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.894562 5025 scope.go:122] "RemoveContainer" containerID="ce49dbc7f7cba4dbab637455ab8130719c4e42ef4cd0688b4012c4e0d7cd1a91" Nov 02 06:45:17.894957 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:17.894929 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n_odh-ai-gateway-infra(9129a2af-4c8c-4aba-a126-dc7af1199ddb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" Nov 02 06:45:17.895966 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.895944 5025 generic.go:356] "Generic (PLEG): container finished" podID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerID="3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879" exitCode=0 Nov 02 06:45:17.896051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.896012 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" event={"ID":"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7","Type":"ContainerDied","Data":"3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879"} Nov 02 06:45:17.896465 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.896446 5025 scope.go:122] "RemoveContainer" containerID="3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879" Nov 02 06:45:17.898985 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.898949 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:45:17.899134 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.899040 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:45:17.899134 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:17.899109 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:45:18.057752 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.057718 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh"] Nov 02 06:45:18.226461 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.226431 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" Nov 02 06:45:18.288092 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.288042 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-maas-api-service-ca\") pod \"7aa1a3bf-856c-4b15-8c4c-1e522da49fff\" (UID: \"7aa1a3bf-856c-4b15-8c4c-1e522da49fff\") " Nov 02 06:45:18.288312 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.288269 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-kube-api-access-dc6jm\" (UniqueName: \"kubernetes.io/projected/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-kube-api-access-dc6jm\") pod \"7aa1a3bf-856c-4b15-8c4c-1e522da49fff\" (UID: \"7aa1a3bf-856c-4b15-8c4c-1e522da49fff\") " Nov 02 06:45:18.288402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.288378 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-maas-api-service-ca" pod "7aa1a3bf-856c-4b15-8c4c-1e522da49fff" (UID: "7aa1a3bf-856c-4b15-8c4c-1e522da49fff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:18.288953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.288928 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-maas-api-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:18.290483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.290457 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-kube-api-access-dc6jm" pod "7aa1a3bf-856c-4b15-8c4c-1e522da49fff" (UID: "7aa1a3bf-856c-4b15-8c4c-1e522da49fff"). InnerVolumeSpecName "kube-api-access-dc6jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:18.390509 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.390466 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc6jm\" (UniqueName: \"kubernetes.io/projected/7aa1a3bf-856c-4b15-8c4c-1e522da49fff-kube-api-access-dc6jm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:18.662350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.662321 5025 scope.go:122] "RemoveContainer" containerID="99c6083792697acdf544735fc66e52e975ef6c88da3b0e8555a72bf62789227f" Nov 02 06:45:18.928872 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.928836 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" Nov 02 06:45:18.929346 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.928863 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh" event={"ID":"7aa1a3bf-856c-4b15-8c4c-1e522da49fff","Type":"ContainerDied","Data":"6742e77a3705a2bf98d317c21dcd87a0062acd1d55166e9cc5b08d198121c68b"} Nov 02 06:45:18.929346 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.929115 5025 scope.go:122] "RemoveContainer" containerID="ef133022f6b2f1d5c08b12be520ed683409ed39e0d7f8d8cb76b7cbfb86c5f9e" Nov 02 06:45:18.934289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.934264 5025 generic.go:356] "Generic (PLEG): container finished" podID="38020a87-e892-4241-8251-c1bff66bac2b" containerID="97faabe31a37c3ccf301e2977f72a83776adc795f112b7cbef9e702cabb9aa76" exitCode=60 Nov 02 06:45:18.934385 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.934338 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" event={"ID":"38020a87-e892-4241-8251-c1bff66bac2b","Type":"ContainerDied","Data":"97faabe31a37c3ccf301e2977f72a83776adc795f112b7cbef9e702cabb9aa76"} Nov 02 06:45:18.934745 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.934727 5025 scope.go:122] "RemoveContainer" containerID="97faabe31a37c3ccf301e2977f72a83776adc795f112b7cbef9e702cabb9aa76" Nov 02 06:45:18.935065 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:18.935041 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb_odh-ai-gateway-infra(38020a87-e892-4241-8251-c1bff66bac2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" podUID="38020a87-e892-4241-8251-c1bff66bac2b" Nov 02 06:45:18.938422 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.938401 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" event={"ID":"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7","Type":"ContainerStarted","Data":"3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65"} Nov 02 06:45:18.941026 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:18.941009 5025 scope.go:122] "RemoveContainer" containerID="035c71b934454315f9d8a0ff6776fcacb403f9aa7ff4759a2dc721f3acbd62b6" Nov 02 06:45:19.020797 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.020761 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n"] Nov 02 06:45:19.226508 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.226058 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh"] Nov 02 06:45:19.231764 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.231738 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-29abe7-29893365-trhxh"] Nov 02 06:45:19.327846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.327819 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" Nov 02 06:45:19.416497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.416459 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9129a2af-4c8c-4aba-a126-dc7af1199ddb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9129a2af-4c8c-4aba-a126-dc7af1199ddb-maas-api-service-ca\") pod \"9129a2af-4c8c-4aba-a126-dc7af1199ddb\" (UID: \"9129a2af-4c8c-4aba-a126-dc7af1199ddb\") " Nov 02 06:45:19.416710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.416546 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9129a2af-4c8c-4aba-a126-dc7af1199ddb-kube-api-access-jl5pt\" (UniqueName: \"kubernetes.io/projected/9129a2af-4c8c-4aba-a126-dc7af1199ddb-kube-api-access-jl5pt\") pod \"9129a2af-4c8c-4aba-a126-dc7af1199ddb\" (UID: \"9129a2af-4c8c-4aba-a126-dc7af1199ddb\") " Nov 02 06:45:19.416781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.416759 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9129a2af-4c8c-4aba-a126-dc7af1199ddb-maas-api-service-ca" pod "9129a2af-4c8c-4aba-a126-dc7af1199ddb" (UID: "9129a2af-4c8c-4aba-a126-dc7af1199ddb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:19.417190 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.417170 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9129a2af-4c8c-4aba-a126-dc7af1199ddb-maas-api-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.418687 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.418662 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9129a2af-4c8c-4aba-a126-dc7af1199ddb-kube-api-access-jl5pt" pod "9129a2af-4c8c-4aba-a126-dc7af1199ddb" (UID: "9129a2af-4c8c-4aba-a126-dc7af1199ddb"). InnerVolumeSpecName "kube-api-access-jl5pt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:19.513819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.513305 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:45:19.513819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.513401 5025 patch_prober.go:29] interesting pod/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 06:45:19.513819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.513470 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 06:45:19.519268 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.519237 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl5pt\" (UniqueName: \"kubernetes.io/projected/9129a2af-4c8c-4aba-a126-dc7af1199ddb-kube-api-access-jl5pt\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.546221 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7135-644a4d"] Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547040 5025 cpu_manager.go:414] "containerMap: removing container" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerName="cleanup" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547056 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerName="cleanup" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547089 5025 cpu_manager.go:414] "containerMap: removing container" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerName="cleanup" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547096 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerName="cleanup" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547128 5025 cpu_manager.go:414] "containerMap: removing container" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerName="cleanup" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547135 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerName="cleanup" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547165 5025 cpu_manager.go:414] "containerMap: removing container" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerName="cleanup" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547172 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerName="cleanup" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547340 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerName="cleanup" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547363 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerName="cleanup" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547388 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerName="cleanup" Nov 02 06:45:19.551097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.547400 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerName="cleanup" Nov 02 06:45:19.556475 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.551964 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-644a4d" Nov 02 06:45:19.616151 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.610571 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 02 06:45:19.616151 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.615304 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 02 06:45:19.616151 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.615383 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-7g9gq\"" Nov 02 06:45:19.620176 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.620144 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-644a4d"] Nov 02 06:45:19.620957 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.620931 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvrl6\" (UniqueName: \"kubernetes.io/projected/e86f56ec-1d57-4631-918d-1672487df3b0-kube-api-access-rvrl6\") pod \"test-curl-7135-644a4d\" (UID: \"e86f56ec-1d57-4631-918d-1672487df3b0\") " pod="opendatahub/test-curl-7135-644a4d" Nov 02 06:45:19.670819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.670784 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" path="/var/lib/kubelet/pods/7aa1a3bf-856c-4b15-8c4c-1e522da49fff/volumes" Nov 02 06:45:19.722170 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.722126 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrl6\" (UniqueName: \"kubernetes.io/projected/e86f56ec-1d57-4631-918d-1672487df3b0-kube-api-access-rvrl6\") pod \"test-curl-7135-644a4d\" (UID: \"e86f56ec-1d57-4631-918d-1672487df3b0\") " pod="opendatahub/test-curl-7135-644a4d" Nov 02 06:45:19.777821 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.777747 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrl6\" (UniqueName: \"kubernetes.io/projected/e86f56ec-1d57-4631-918d-1672487df3b0-kube-api-access-rvrl6\") pod \"test-curl-7135-644a4d\" (UID: \"e86f56ec-1d57-4631-918d-1672487df3b0\") " pod="opendatahub/test-curl-7135-644a4d" Nov 02 06:45:19.869392 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.869355 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-644a4d" Nov 02 06:45:19.952007 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.951977 5025 generic.go:356] "Generic (PLEG): container finished" podID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerID="0a9ec1087795a279741551267e551bb0a6e16e863f0744767c435268eca84880" exitCode=60 Nov 02 06:45:19.952470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.952192 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" event={"ID":"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb","Type":"ContainerDied","Data":"0a9ec1087795a279741551267e551bb0a6e16e863f0744767c435268eca84880"} Nov 02 06:45:19.952470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.952257 5025 scope.go:122] "RemoveContainer" containerID="99c6083792697acdf544735fc66e52e975ef6c88da3b0e8555a72bf62789227f" Nov 02 06:45:19.952765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.952746 5025 scope.go:122] "RemoveContainer" containerID="0a9ec1087795a279741551267e551bb0a6e16e863f0744767c435268eca84880" Nov 02 06:45:19.953602 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:19.953137 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5_odh-ai-gateway-infra(b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" Nov 02 06:45:19.955042 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.954963 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" Nov 02 06:45:19.955042 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:19.954973 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n" event={"ID":"9129a2af-4c8c-4aba-a126-dc7af1199ddb","Type":"ContainerDied","Data":"cd88f674ec80990f4c2f1bad4592333d9d5450cc6b55893aea8a0627b2bf5540"} Nov 02 06:45:20.023127 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.021122 5025 scope.go:122] "RemoveContainer" containerID="ce49dbc7f7cba4dbab637455ab8130719c4e42ef4cd0688b4012c4e0d7cd1a91" Nov 02 06:45:20.118237 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.118189 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb"] Nov 02 06:45:20.181011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.180979 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n"] Nov 02 06:45:20.216248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.215544 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893365-w4g7n"] Nov 02 06:45:20.315153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.313330 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7135-644a4d"] Nov 02 06:45:20.316113 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:45:20.316022 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode86f56ec_1d57_4631_918d_1672487df3b0.slice/crio-b6c1e18cdf0cb4e43f4f998b69b067e2d4947b01dc3d79744cd0d7b0bd179cb8 WatchSource:0}: Error finding container b6c1e18cdf0cb4e43f4f998b69b067e2d4947b01dc3d79744cd0d7b0bd179cb8: Status 404 returned error can't find the container with id b6c1e18cdf0cb4e43f4f998b69b067e2d4947b01dc3d79744cd0d7b0bd179cb8 Nov 02 06:45:20.432928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.432901 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" Nov 02 06:45:20.504823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.504702 5025 patch_prober.go:29] interesting pod/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 06:45:20.504823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.504770 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 06:45:20.534977 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.534924 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38020a87-e892-4241-8251-c1bff66bac2b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38020a87-e892-4241-8251-c1bff66bac2b-maas-api-service-ca\") pod \"38020a87-e892-4241-8251-c1bff66bac2b\" (UID: \"38020a87-e892-4241-8251-c1bff66bac2b\") " Nov 02 06:45:20.535234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.535212 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38020a87-e892-4241-8251-c1bff66bac2b-kube-api-access-t7954\" (UniqueName: \"kubernetes.io/projected/38020a87-e892-4241-8251-c1bff66bac2b-kube-api-access-t7954\") pod \"38020a87-e892-4241-8251-c1bff66bac2b\" (UID: \"38020a87-e892-4241-8251-c1bff66bac2b\") " Nov 02 06:45:20.535341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.535318 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38020a87-e892-4241-8251-c1bff66bac2b-maas-api-service-ca" pod "38020a87-e892-4241-8251-c1bff66bac2b" (UID: "38020a87-e892-4241-8251-c1bff66bac2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:20.535891 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.535871 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38020a87-e892-4241-8251-c1bff66bac2b-maas-api-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:20.537457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.537434 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38020a87-e892-4241-8251-c1bff66bac2b-kube-api-access-t7954" pod "38020a87-e892-4241-8251-c1bff66bac2b" (UID: "38020a87-e892-4241-8251-c1bff66bac2b"). InnerVolumeSpecName "kube-api-access-t7954". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:20.637881 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.637783 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7954\" (UniqueName: \"kubernetes.io/projected/38020a87-e892-4241-8251-c1bff66bac2b-kube-api-access-t7954\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:20.967716 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.967678 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-644a4d" event={"ID":"e86f56ec-1d57-4631-918d-1672487df3b0","Type":"ContainerStarted","Data":"edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f"} Nov 02 06:45:20.967716 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.967724 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7135-644a4d" event={"ID":"e86f56ec-1d57-4631-918d-1672487df3b0","Type":"ContainerStarted","Data":"b6c1e18cdf0cb4e43f4f998b69b067e2d4947b01dc3d79744cd0d7b0bd179cb8"} Nov 02 06:45:20.971550 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.971525 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" event={"ID":"38020a87-e892-4241-8251-c1bff66bac2b","Type":"ContainerDied","Data":"ac423f1eb675342ce927cd26d4c10b6662642566db50e5242e96d666f9f98665"} Nov 02 06:45:20.971685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.971573 5025 scope.go:122] "RemoveContainer" containerID="97faabe31a37c3ccf301e2977f72a83776adc795f112b7cbef9e702cabb9aa76" Nov 02 06:45:20.971685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:20.971534 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb" Nov 02 06:45:21.081638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.081573 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7135-644a4d" podStartSLOduration=2.081558715 podStartE2EDuration="2.081558715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:21.077919027 +0000 UTC m=+1697.526295783" watchObservedRunningTime="2026-11-02 06:45:21.081558715 +0000 UTC m=+1697.529935471" Nov 02 06:45:21.116093 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.113984 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5"] Nov 02 06:45:21.223819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.221169 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb"] Nov 02 06:45:21.228974 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.227799 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-dcf9c3-29893365-6gcbb"] Nov 02 06:45:21.321525 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.321496 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" Nov 02 06:45:21.449191 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.449105 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-kube-api-access-5tczm\" (UniqueName: \"kubernetes.io/projected/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-kube-api-access-5tczm\") pod \"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb\" (UID: \"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb\") " Nov 02 06:45:21.449340 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.449272 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-maas-api-service-ca\") pod \"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb\" (UID: \"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb\") " Nov 02 06:45:21.449719 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.449688 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-maas-api-service-ca" pod "b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" (UID: "b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:21.450147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.450123 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-maas-api-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:21.451617 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.451588 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-kube-api-access-5tczm" pod "b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" (UID: "b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb"). InnerVolumeSpecName "kube-api-access-5tczm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:21.508533 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.508497 5025 patch_prober.go:29] interesting pod/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 06:45:21.508727 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.508559 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 06:45:21.552216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.552176 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tczm\" (UniqueName: \"kubernetes.io/projected/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb-kube-api-access-5tczm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:21.673351 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.673313 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38020a87-e892-4241-8251-c1bff66bac2b" path="/var/lib/kubelet/pods/38020a87-e892-4241-8251-c1bff66bac2b/volumes" Nov 02 06:45:21.673966 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.673940 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" path="/var/lib/kubelet/pods/9129a2af-4c8c-4aba-a126-dc7af1199ddb/volumes" Nov 02 06:45:21.982589 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.982503 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" Nov 02 06:45:21.982589 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.982538 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5" event={"ID":"b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb","Type":"ContainerDied","Data":"4f282df32e5335c52e162031a95b1e52ffa86ce3afeac2f991e55ca28ab902fa"} Nov 02 06:45:21.983112 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:21.982599 5025 scope.go:122] "RemoveContainer" containerID="0a9ec1087795a279741551267e551bb0a6e16e863f0744767c435268eca84880" Nov 02 06:45:22.192101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:22.188679 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:22.254251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:22.254139 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5"] Nov 02 06:45:22.270158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:22.270030 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a25b83-29893365-9jwp5"] Nov 02 06:45:22.495102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:22.494113 5025 patch_prober.go:29] interesting pod/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 06:45:22.495102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:22.494174 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 06:45:22.542772 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:22.542687 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7135-644a4d"] Nov 02 06:45:22.547262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:22.547234 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7135-644a4d"] Nov 02 06:45:23.015317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:23.015283 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7135-644a4d" podUID="e86f56ec-1d57-4631-918d-1672487df3b0" containerName="test-curl-7135-644a4d" containerID="cri-o://edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f" gracePeriod=2 Nov 02 06:45:23.039354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:23.039325 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:23.064265 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:23.064228 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-644a4d\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="e86f56ec-1d57-4631-918d-1672487df3b0" pod="opendatahub/test-curl-7135-644a4d" Nov 02 06:45:23.494101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:23.493114 5025 patch_prober.go:29] interesting pod/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 02 06:45:23.494101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:23.493179 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 02 06:45:23.671221 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:23.671187 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" path="/var/lib/kubelet/pods/b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb/volumes" Nov 02 06:45:23.714176 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:23.714136 5025 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7135-644a4d\" is forbidden: User \"system:node:ip-10-0-130-34.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-34.us-west-2.compute.internal' and this object" podUID="e86f56ec-1d57-4631-918d-1672487df3b0" pod="opendatahub/test-curl-7135-644a4d" Nov 02 06:45:24.493450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:24.493413 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:45:24.493987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:24.493962 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:45:24.494781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:24.494760 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:45:25.276342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:25.276281 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-644a4d" Nov 02 06:45:25.335149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:25.335114 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86f56ec-1d57-4631-918d-1672487df3b0-kube-api-access-rvrl6\" (UniqueName: \"kubernetes.io/projected/e86f56ec-1d57-4631-918d-1672487df3b0-kube-api-access-rvrl6\") pod \"e86f56ec-1d57-4631-918d-1672487df3b0\" (UID: \"e86f56ec-1d57-4631-918d-1672487df3b0\") " Nov 02 06:45:25.337236 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:25.337207 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86f56ec-1d57-4631-918d-1672487df3b0-kube-api-access-rvrl6" pod "e86f56ec-1d57-4631-918d-1672487df3b0" (UID: "e86f56ec-1d57-4631-918d-1672487df3b0"). InnerVolumeSpecName "kube-api-access-rvrl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:25.437304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:25.437263 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvrl6\" (UniqueName: \"kubernetes.io/projected/e86f56ec-1d57-4631-918d-1672487df3b0-kube-api-access-rvrl6\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:25.669776 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:25.669748 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86f56ec-1d57-4631-918d-1672487df3b0" path="/var/lib/kubelet/pods/e86f56ec-1d57-4631-918d-1672487df3b0/volumes" Nov 02 06:45:26.038691 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:26.038602 5025 generic.go:356] "Generic (PLEG): container finished" podID="e86f56ec-1d57-4631-918d-1672487df3b0" containerID="edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f" exitCode=137 Nov 02 06:45:26.038691 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:26.038644 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7135-644a4d" Nov 02 06:45:26.038891 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:26.038706 5025 scope.go:122] "RemoveContainer" containerID="edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f" Nov 02 06:45:26.048564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:26.048535 5025 scope.go:122] "RemoveContainer" containerID="edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f" Nov 02 06:45:26.048820 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:26.048801 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f\": container with ID starting with edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f not found: ID does not exist" containerID="edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f" Nov 02 06:45:26.048866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:26.048830 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f"} err="failed to get container status \"edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f\": rpc error: code = NotFound desc = could not find container \"edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f\": container with ID starting with edf94322cfd9f7759cd300e48d1a1151d73b43252093937265a03213f33b276f not found: ID does not exist" Nov 02 06:45:26.995745 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:26.995712 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:31.921024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:31.920992 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:32.709258 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:32.709224 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:45:34.854176 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:34.854145 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p"] Nov 02 06:45:35.243047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:35.243004 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:35.439765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:35.439728 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:37.222100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:37.213648 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:38.728340 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:38.728287 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:39.386099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:39.386042 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7"] Nov 02 06:45:39.487458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:39.487423 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t"] Nov 02 06:45:39.523785 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:39.523753 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:39.776217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:39.776169 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:39.855439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:39.855398 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" podUID="b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1" containerName="payload-processing" containerID="cri-o://72a66ff18f7efe1f77e2410eeb3d0fb2fc95c828ebf4c651217b483fb6648de6" gracePeriod=25 Nov 02 06:45:39.970714 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:39.970672 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:40.400162 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.400128 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj"] Nov 02 06:45:40.400885 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.400865 5025 cpu_manager.go:414] "containerMap: removing container" podUID="e86f56ec-1d57-4631-918d-1672487df3b0" containerName="test-curl-7135-644a4d" Nov 02 06:45:40.400885 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.400885 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86f56ec-1d57-4631-918d-1672487df3b0" containerName="test-curl-7135-644a4d" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.400904 5025 cpu_manager.go:414] "containerMap: removing container" podUID="38020a87-e892-4241-8251-c1bff66bac2b" containerName="cleanup" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.400912 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38020a87-e892-4241-8251-c1bff66bac2b" containerName="cleanup" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.400932 5025 cpu_manager.go:414] "containerMap: removing container" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerName="cleanup" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.400940 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerName="cleanup" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.400970 5025 cpu_manager.go:414] "containerMap: removing container" podUID="38020a87-e892-4241-8251-c1bff66bac2b" containerName="cleanup" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.400978 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38020a87-e892-4241-8251-c1bff66bac2b" containerName="cleanup" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401003 5025 cpu_manager.go:414] "containerMap: removing container" podUID="38020a87-e892-4241-8251-c1bff66bac2b" containerName="cleanup" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401010 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38020a87-e892-4241-8251-c1bff66bac2b" containerName="cleanup" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401021 5025 cpu_manager.go:414] "containerMap: removing container" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerName="cleanup" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401029 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerName="cleanup" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401044 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerName="cleanup" Nov 02 06:45:40.401060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401052 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerName="cleanup" Nov 02 06:45:40.401685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401067 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerName="cleanup" Nov 02 06:45:40.401685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401089 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerName="cleanup" Nov 02 06:45:40.401685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401120 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerName="cleanup" Nov 02 06:45:40.401685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401127 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerName="cleanup" Nov 02 06:45:40.401685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401300 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="38020a87-e892-4241-8251-c1bff66bac2b" containerName="cleanup" Nov 02 06:45:40.401685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401315 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerName="cleanup" Nov 02 06:45:40.401685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401325 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="9129a2af-4c8c-4aba-a126-dc7af1199ddb" containerName="cleanup" Nov 02 06:45:40.401685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401339 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerName="cleanup" Nov 02 06:45:40.401685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401354 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86f56ec-1d57-4631-918d-1672487df3b0" containerName="test-curl-7135-644a4d" Nov 02 06:45:40.401685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401367 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa1a3bf-856c-4b15-8c4c-1e522da49fff" containerName="cleanup" Nov 02 06:45:40.401685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.401379 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="38020a87-e892-4241-8251-c1bff66bac2b" containerName="cleanup" Nov 02 06:45:40.406839 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.406818 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.456640 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.456600 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-37b53e-openshift-default-dockercfg-xtxj7\"" Nov 02 06:45:40.497535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.497112 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istiod-ca-cert\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.497535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.497164 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-data\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.497535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.497343 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-token\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.497535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.497394 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-podinfo\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.497535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.497432 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-certs\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.497535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.497478 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-credential-socket\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.498032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.497620 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-socket\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.498032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.497652 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-envoy\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.498032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.497700 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttv9z\" (UniqueName: \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-kube-api-access-ttv9z\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.504225 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.504198 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj"] Nov 02 06:45:40.588105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.579682 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:40.695505 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.693852 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-socket\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.697177 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.693926 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-envoy\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.697522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.697229 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttv9z\" (UniqueName: \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-kube-api-access-ttv9z\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.697634 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.697616 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istiod-ca-cert\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.697811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.697782 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-data\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.698005 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.697987 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-token\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.698210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.698190 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-podinfo\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.698348 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.698249 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-certs\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.698508 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.698393 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-credential-socket\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.700678 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.700547 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-credential-socket\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.701296 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.701104 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istiod-ca-cert\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.702048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.702024 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-socket\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.702651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.702437 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-data\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.702726 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.702566 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-certs\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.712335 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.712305 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-envoy\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.713325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.713265 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-podinfo\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.755311 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.755276 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-token\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.757875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.757843 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttv9z\" (UniqueName: \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-kube-api-access-ttv9z\") pod \"e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:40.814106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:40.811791 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:41.022175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:41.020866 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:41.686500 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:45:41.686472 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc2f04e7_114a_4ab0_8e5f_1035460391fd.slice/crio-fc7141ae38dda2040bf466026a8997c1199b18e8ad613cedf14f008542896419 WatchSource:0}: Error finding container fc7141ae38dda2040bf466026a8997c1199b18e8ad613cedf14f008542896419: Status 404 returned error can't find the container with id fc7141ae38dda2040bf466026a8997c1199b18e8ad613cedf14f008542896419 Nov 02 06:45:41.686740 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:41.686720 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj"] Nov 02 06:45:41.689201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:41.689165 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:45:41.689301 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:41.689244 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:45:41.689301 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:41.689293 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:45:42.249406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:42.249364 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" event={"ID":"bc2f04e7-114a-4ab0-8e5f-1035460391fd","Type":"ContainerStarted","Data":"56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f"} Nov 02 06:45:42.249893 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:42.249428 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" event={"ID":"bc2f04e7-114a-4ab0-8e5f-1035460391fd","Type":"ContainerStarted","Data":"fc7141ae38dda2040bf466026a8997c1199b18e8ad613cedf14f008542896419"} Nov 02 06:45:42.338317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:42.338240 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:42.410826 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:42.410773 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" podStartSLOduration=2.410757538 podStartE2EDuration="2.410757538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:42.409540268 +0000 UTC m=+1718.857917044" watchObservedRunningTime="2026-11-02 06:45:42.410757538 +0000 UTC m=+1718.859134293" Nov 02 06:45:43.021310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:43.021279 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:43.027766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:43.027740 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:43.256111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:43.256053 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:43.257062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:43.257040 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:45:43.336924 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:43.336844 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" podUID="b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:45:44.386787 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.386700 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" podUID="02bbf1c2-fc80-4ec0-98b5-00258244e5ef" containerName="payload-processing" containerID="cri-o://fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b" gracePeriod=25 Nov 02 06:45:44.488804 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.488765 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" podUID="8b284776-73cc-4707-a5ea-b074892fc599" containerName="payload-pre-processing" containerID="cri-o://8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc" gracePeriod=25 Nov 02 06:45:44.681820 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.681793 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:45:44.717563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.717532 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-kube-api-access-m9t4g\" (UniqueName: \"kubernetes.io/projected/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-kube-api-access-m9t4g\") pod \"02bbf1c2-fc80-4ec0-98b5-00258244e5ef\" (UID: \"02bbf1c2-fc80-4ec0-98b5-00258244e5ef\") " Nov 02 06:45:44.717734 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.717608 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-plugins-config-volume\") pod \"02bbf1c2-fc80-4ec0-98b5-00258244e5ef\" (UID: \"02bbf1c2-fc80-4ec0-98b5-00258244e5ef\") " Nov 02 06:45:44.717991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.717965 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-plugins-config-volume" pod "02bbf1c2-fc80-4ec0-98b5-00258244e5ef" (UID: "02bbf1c2-fc80-4ec0-98b5-00258244e5ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:44.718869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.718845 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:44.719874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.719846 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-kube-api-access-m9t4g" pod "02bbf1c2-fc80-4ec0-98b5-00258244e5ef" (UID: "02bbf1c2-fc80-4ec0-98b5-00258244e5ef"). InnerVolumeSpecName "kube-api-access-m9t4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:44.777709 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.777686 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:45:44.819878 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.819847 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b284776-73cc-4707-a5ea-b074892fc599-kube-api-access-9p5d6\" (UniqueName: \"kubernetes.io/projected/8b284776-73cc-4707-a5ea-b074892fc599-kube-api-access-9p5d6\") pod \"8b284776-73cc-4707-a5ea-b074892fc599\" (UID: \"8b284776-73cc-4707-a5ea-b074892fc599\") " Nov 02 06:45:44.820021 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.819911 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b284776-73cc-4707-a5ea-b074892fc599-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b284776-73cc-4707-a5ea-b074892fc599-plugins-config-volume\") pod \"8b284776-73cc-4707-a5ea-b074892fc599\" (UID: \"8b284776-73cc-4707-a5ea-b074892fc599\") " Nov 02 06:45:44.820338 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.820316 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b284776-73cc-4707-a5ea-b074892fc599-plugins-config-volume" pod "8b284776-73cc-4707-a5ea-b074892fc599" (UID: "8b284776-73cc-4707-a5ea-b074892fc599"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:45:44.820801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.820775 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9t4g\" (UniqueName: \"kubernetes.io/projected/02bbf1c2-fc80-4ec0-98b5-00258244e5ef-kube-api-access-m9t4g\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:44.820861 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.820808 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b284776-73cc-4707-a5ea-b074892fc599-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:44.821891 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.821858 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b284776-73cc-4707-a5ea-b074892fc599-kube-api-access-9p5d6" pod "8b284776-73cc-4707-a5ea-b074892fc599" (UID: "8b284776-73cc-4707-a5ea-b074892fc599"). InnerVolumeSpecName "kube-api-access-9p5d6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:45:44.922262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:44.922225 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p5d6\" (UniqueName: \"kubernetes.io/projected/8b284776-73cc-4707-a5ea-b074892fc599-kube-api-access-9p5d6\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:45:45.262140 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.262036 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:45.273937 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.273902 5025 generic.go:356] "Generic (PLEG): container finished" podID="02bbf1c2-fc80-4ec0-98b5-00258244e5ef" containerID="fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b" exitCode=0 Nov 02 06:45:45.274123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.273950 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" Nov 02 06:45:45.274123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.273987 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" event={"ID":"02bbf1c2-fc80-4ec0-98b5-00258244e5ef","Type":"ContainerDied","Data":"fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b"} Nov 02 06:45:45.274123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.274041 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7" event={"ID":"02bbf1c2-fc80-4ec0-98b5-00258244e5ef","Type":"ContainerDied","Data":"d30eb0b04ff2034a6b208f5655b7dd7c1fdd5af6556224c769098652eeaac63f"} Nov 02 06:45:45.274123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.274070 5025 scope.go:122] "RemoveContainer" containerID="fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b" Nov 02 06:45:45.275627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.275608 5025 generic.go:356] "Generic (PLEG): container finished" podID="8b284776-73cc-4707-a5ea-b074892fc599" containerID="8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc" exitCode=0 Nov 02 06:45:45.275742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.275653 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" Nov 02 06:45:45.275742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.275678 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" event={"ID":"8b284776-73cc-4707-a5ea-b074892fc599","Type":"ContainerDied","Data":"8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc"} Nov 02 06:45:45.275853 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.275741 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t" event={"ID":"8b284776-73cc-4707-a5ea-b074892fc599","Type":"ContainerDied","Data":"c1652a926962bc47085e444dfe7e71ba092cd995aa2540bee7e258814984d641"} Nov 02 06:45:45.284861 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.284830 5025 scope.go:122] "RemoveContainer" containerID="fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b" Nov 02 06:45:45.285153 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:45.285127 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b\": container with ID starting with fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b not found: ID does not exist" containerID="fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b" Nov 02 06:45:45.285250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.285162 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b"} err="failed to get container status \"fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b\": rpc error: code = NotFound desc = could not find container \"fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b\": container with ID starting with fd13294bf5728951af7a9a167b60298e5407cd0bf794239cf58fd89f0fcfdb8b not found: ID does not exist" Nov 02 06:45:45.285250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.285179 5025 scope.go:122] "RemoveContainer" containerID="8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc" Nov 02 06:45:45.294347 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.294327 5025 scope.go:122] "RemoveContainer" containerID="8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc" Nov 02 06:45:45.294609 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:45:45.294587 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc\": container with ID starting with 8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc not found: ID does not exist" containerID="8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc" Nov 02 06:45:45.294663 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.294615 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc"} err="failed to get container status \"8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc\": rpc error: code = NotFound desc = could not find container \"8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc\": container with ID starting with 8a9925db9707912997c4aa5d4b734d85f76902b43b1a12c7bc9e4aa937dbe6cc not found: ID does not exist" Nov 02 06:45:45.434266 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.434233 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7"] Nov 02 06:45:45.455455 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.455422 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:45.481163 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.481133 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-e5502564-67bd495475-xtbx7"] Nov 02 06:45:45.599021 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.598932 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t"] Nov 02 06:45:45.604143 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.604118 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-e5502564-859c958dcf-6pd7t"] Nov 02 06:45:45.670193 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.670165 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02bbf1c2-fc80-4ec0-98b5-00258244e5ef" path="/var/lib/kubelet/pods/02bbf1c2-fc80-4ec0-98b5-00258244e5ef/volumes" Nov 02 06:45:45.670659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:45.670637 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b284776-73cc-4707-a5ea-b074892fc599" path="/var/lib/kubelet/pods/8b284776-73cc-4707-a5ea-b074892fc599/volumes" Nov 02 06:45:47.513163 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:47.513058 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:48.363142 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.363111 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:48.559094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.558993 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:48.920095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.920037 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r"] Nov 02 06:45:48.921541 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.921512 5025 cpu_manager.go:414] "containerMap: removing container" podUID="8b284776-73cc-4707-a5ea-b074892fc599" containerName="payload-pre-processing" Nov 02 06:45:48.921750 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.921732 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b284776-73cc-4707-a5ea-b074892fc599" containerName="payload-pre-processing" Nov 02 06:45:48.921879 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.921865 5025 cpu_manager.go:414] "containerMap: removing container" podUID="02bbf1c2-fc80-4ec0-98b5-00258244e5ef" containerName="payload-processing" Nov 02 06:45:48.921986 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.921973 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02bbf1c2-fc80-4ec0-98b5-00258244e5ef" containerName="payload-processing" Nov 02 06:45:48.922332 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.922312 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="38020a87-e892-4241-8251-c1bff66bac2b" containerName="cleanup" Nov 02 06:45:48.922457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.922445 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b284776-73cc-4707-a5ea-b074892fc599" containerName="payload-pre-processing" Nov 02 06:45:48.922556 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.922544 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3f8a1f2-caa0-4b2a-be5f-c92187dfdbbb" containerName="cleanup" Nov 02 06:45:48.922638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.922628 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="02bbf1c2-fc80-4ec0-98b5-00258244e5ef" containerName="payload-processing" Nov 02 06:45:48.928543 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.928525 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r"] Nov 02 06:45:48.928765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.928751 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:45:48.984507 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.984471 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-37b53e-dockercfg-ghlvn\"" Nov 02 06:45:48.986113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:48.985482 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-37b53e\"" Nov 02 06:45:49.130146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:49.112375 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r\" (UID: \"ea75ec12-0c13-4be6-ad9d-7cfa0f176780\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:45:49.130146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:49.112567 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhx6\" (UniqueName: \"kubernetes.io/projected/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-kube-api-access-zlhx6\") pod \"payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r\" (UID: \"ea75ec12-0c13-4be6-ad9d-7cfa0f176780\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:45:49.214610 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:49.214518 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r\" (UID: \"ea75ec12-0c13-4be6-ad9d-7cfa0f176780\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:45:49.214781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:49.214705 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhx6\" (UniqueName: \"kubernetes.io/projected/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-kube-api-access-zlhx6\") pod \"payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r\" (UID: \"ea75ec12-0c13-4be6-ad9d-7cfa0f176780\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:45:49.215482 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:49.215461 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r\" (UID: \"ea75ec12-0c13-4be6-ad9d-7cfa0f176780\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:45:49.269052 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:49.269017 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhx6\" (UniqueName: \"kubernetes.io/projected/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-kube-api-access-zlhx6\") pod \"payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r\" (UID: \"ea75ec12-0c13-4be6-ad9d-7cfa0f176780\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:45:49.295191 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:49.295158 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:45:49.731094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:49.731051 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r"] Nov 02 06:45:49.731764 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:45:49.731740 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea75ec12_0c13_4be6_ad9d_7cfa0f176780.slice/crio-77635f51ba3157315eea3a58528a18730e5402b2bdf27c71ecd425df7bba6733 WatchSource:0}: Error finding container 77635f51ba3157315eea3a58528a18730e5402b2bdf27c71ecd425df7bba6733: Status 404 returned error can't find the container with id 77635f51ba3157315eea3a58528a18730e5402b2bdf27c71ecd425df7bba6733 Nov 02 06:45:50.316562 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:50.316467 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" event={"ID":"ea75ec12-0c13-4be6-ad9d-7cfa0f176780","Type":"ContainerStarted","Data":"f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352"} Nov 02 06:45:50.316562 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:50.316527 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" event={"ID":"ea75ec12-0c13-4be6-ad9d-7cfa0f176780","Type":"ContainerStarted","Data":"77635f51ba3157315eea3a58528a18730e5402b2bdf27c71ecd425df7bba6733"} Nov 02 06:45:50.316562 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:50.316558 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:45:50.465106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:50.444536 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" podStartSLOduration=2.444514787 podStartE2EDuration="2.444514787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:45:50.4395358 +0000 UTC m=+1726.887912580" watchObservedRunningTime="2026-11-02 06:45:50.444514787 +0000 UTC m=+1726.892891543" Nov 02 06:45:52.513100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:52.504585 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:53.337637 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:53.337598 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" podUID="b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 06:45:57.279586 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:57.269127 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:59.292269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:59.292228 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:45:59.701278 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:45:59.701234 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:00.155209 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.155152 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v"] Nov 02 06:46:00.439358 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.387760 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:00.467507 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.467467 5025 generic.go:356] "Generic (PLEG): container finished" podID="b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1" containerID="72a66ff18f7efe1f77e2410eeb3d0fb2fc95c828ebf4c651217b483fb6648de6" exitCode=0 Nov 02 06:46:00.476526 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.476489 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" event={"ID":"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1","Type":"ContainerDied","Data":"72a66ff18f7efe1f77e2410eeb3d0fb2fc95c828ebf4c651217b483fb6648de6"} Nov 02 06:46:00.476853 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.476834 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.481485 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.481461 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v"] Nov 02 06:46:00.556248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.555705 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-envoy\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.556248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.555755 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-data\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.556248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.555834 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-token\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.556248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.555872 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68a4b47-40b0-493f-a07e-721871e27a0f-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.556248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.555902 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-podinfo\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.556248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.555934 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-socket\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.556248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.556019 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-certs\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.581453 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.561745 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66lh9\" (UniqueName: \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-kube-api-access-66lh9\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.581453 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.561842 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-credential-socket\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.616545 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.616511 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-6d13c3-openshift-default-dockercfg-b5bsm\"" Nov 02 06:46:00.665842 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.665426 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:46:00.667138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.666385 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-certs\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.667138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.666442 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66lh9\" (UniqueName: \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-kube-api-access-66lh9\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.667138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.666512 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-credential-socket\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.667138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.666743 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-envoy\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.667138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.666771 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-data\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.667138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.666870 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-token\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.667138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.666910 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68a4b47-40b0-493f-a07e-721871e27a0f-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.667138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.666940 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-podinfo\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.667138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.666986 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-socket\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.667704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.667595 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-socket\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.667941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.667892 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-certs\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.668650 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.668629 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-credential-socket\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.670854 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.669575 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-data\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.673206 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.673182 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-envoy\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.675041 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.675019 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-podinfo\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.677128 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.675719 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68a4b47-40b0-493f-a07e-721871e27a0f-istiod-ca-cert\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.725171 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.725143 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66lh9\" (UniqueName: \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-kube-api-access-66lh9\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.726466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.726440 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-token\") pod \"e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.858958 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.858918 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:00.890025 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.889991 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-kube-api-access-krjhq\" (UniqueName: \"kubernetes.io/projected/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-kube-api-access-krjhq\") pod \"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1\" (UID: \"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1\") " Nov 02 06:46:00.890230 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.890219 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-plugins-config-volume\") pod \"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1\" (UID: \"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1\") " Nov 02 06:46:00.890666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.890636 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-plugins-config-volume" pod "b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1" (UID: "b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:00.891225 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.891202 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:00.892781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.892751 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-kube-api-access-krjhq" pod "b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1" (UID: "b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1"). InnerVolumeSpecName "kube-api-access-krjhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:00.965230 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.965196 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:00.993574 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:00.993528 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krjhq\" (UniqueName: \"kubernetes.io/projected/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1-kube-api-access-krjhq\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:01.327223 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.327192 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:46:01.408964 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:46:01.408920 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc68a4b47_40b0_493f_a07e_721871e27a0f.slice/crio-ceea010347181beafc3d84b674c4bf06ee8d7ee0dfaf0bf5f966fc3c119c7653 WatchSource:0}: Error finding container ceea010347181beafc3d84b674c4bf06ee8d7ee0dfaf0bf5f966fc3c119c7653: Status 404 returned error can't find the container with id ceea010347181beafc3d84b674c4bf06ee8d7ee0dfaf0bf5f966fc3c119c7653 Nov 02 06:46:01.411573 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.411539 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:46:01.411684 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.411610 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:46:01.411684 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.411649 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:46:01.428927 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.428904 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v"] Nov 02 06:46:01.478982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.478945 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" event={"ID":"c68a4b47-40b0-493f-a07e-721871e27a0f","Type":"ContainerStarted","Data":"ceea010347181beafc3d84b674c4bf06ee8d7ee0dfaf0bf5f966fc3c119c7653"} Nov 02 06:46:01.480852 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.480827 5025 generic.go:356] "Generic (PLEG): container finished" podID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerID="944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c" exitCode=0 Nov 02 06:46:01.481189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.480906 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" event={"ID":"288d008c-009e-4543-ae54-4ef60cc18c1a","Type":"ContainerDied","Data":"944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c"} Nov 02 06:46:01.481472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.481454 5025 scope.go:122] "RemoveContainer" containerID="944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c" Nov 02 06:46:01.482362 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.482343 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" event={"ID":"b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1","Type":"ContainerDied","Data":"d70ee041a6ba9fea8e2bb0d02d798046d866e6c93585a32bdc74b2d9292ec7c9"} Nov 02 06:46:01.482421 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.482387 5025 scope.go:122] "RemoveContainer" containerID="72a66ff18f7efe1f77e2410eeb3d0fb2fc95c828ebf4c651217b483fb6648de6" Nov 02 06:46:01.482527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.482512 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p" Nov 02 06:46:01.483778 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.483745 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:46:01.483857 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.483838 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:46:01.483926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.483911 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:46:01.723557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.723509 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p"] Nov 02 06:46:01.726859 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:01.726832 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4pw2p"] Nov 02 06:46:02.549403 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:02.545721 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:02.563708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:02.563671 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" event={"ID":"c68a4b47-40b0-493f-a07e-721871e27a0f","Type":"ContainerStarted","Data":"356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8"} Nov 02 06:46:02.619099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:02.618942 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" event={"ID":"288d008c-009e-4543-ae54-4ef60cc18c1a","Type":"ContainerStarted","Data":"96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416"} Nov 02 06:46:02.714034 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:02.713982 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" podStartSLOduration=2.71396718 podStartE2EDuration="2.71396718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:46:02.710468087 +0000 UTC m=+1739.158844844" watchObservedRunningTime="2026-11-02 06:46:02.71396718 +0000 UTC m=+1739.162343936" Nov 02 06:46:02.733636 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:46:02.733602 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:46:02.966100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:02.966033 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:02.971787 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:02.971759 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:03.047645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:03.047606 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:46:03.048890 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:03.048865 5025 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 06:46:03.049013 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:03.048928 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 06:46:03.631410 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:03.631382 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:03.632282 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:03.632257 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:46:03.680627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:03.680596 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1" path="/var/lib/kubelet/pods/b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1/volumes" Nov 02 06:46:04.048170 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:04.048136 5025 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 06:46:04.048341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:04.048187 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 06:46:04.911070 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:04.911029 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj"] Nov 02 06:46:04.911620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:04.911352 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" containerID="cri-o://6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3" gracePeriod=30 Nov 02 06:46:05.049913 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:05.049447 5025 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 06:46:05.049913 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:05.049606 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 06:46:05.476840 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:05.476793 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:05.677298 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:05.677261 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn"] Nov 02 06:46:05.677528 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:05.677507 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" podUID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" containerName="istio-proxy" containerID="cri-o://2528049797f3ae3042222247fbd3a3aadbf183d95a86f12cb7e3c4f3af5bb617" gracePeriod=30 Nov 02 06:46:05.728378 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:05.728262 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:06.047904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:06.047818 5025 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 06:46:06.047904 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:06.047873 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 06:46:07.048243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:07.048202 5025 patch_prober.go:29] interesting pod/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body= Nov 02 06:46:07.048682 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:07.048267 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" Nov 02 06:46:07.559102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:07.548841 5025 patch_prober.go:29] interesting pod/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:46:07.559102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:07.548901 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" podUID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:46:07.570366 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:07.565911 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:08.052536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:08.052507 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:46:08.053025 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:08.053004 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:46:08.053812 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:08.053797 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 06:46:08.559345 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:08.559315 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:08.863578 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:08.863425 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:09.129147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.123883 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl"] Nov 02 06:46:09.129147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.125114 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1" containerName="payload-processing" Nov 02 06:46:09.129147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.125137 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1" containerName="payload-processing" Nov 02 06:46:09.129147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.125300 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8ea7dfa-3a92-4e0c-a974-06815bcb3ff1" containerName="payload-processing" Nov 02 06:46:09.167884 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.167852 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:46:09.181830 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.180693 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl"] Nov 02 06:46:09.243148 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.232400 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-6d13c3\"" Nov 02 06:46:09.243148 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.233283 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-6d13c3-dockercfg-xfltd\"" Nov 02 06:46:09.296208 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.296151 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvptn\" (UniqueName: \"kubernetes.io/projected/dfc762bd-184c-428f-b6e4-7fd135b57154-kube-api-access-fvptn\") pod \"payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl\" (UID: \"dfc762bd-184c-428f-b6e4-7fd135b57154\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:46:09.296406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.296389 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc762bd-184c-428f-b6e4-7fd135b57154-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl\" (UID: \"dfc762bd-184c-428f-b6e4-7fd135b57154\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:46:09.417777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.417744 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc762bd-184c-428f-b6e4-7fd135b57154-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl\" (UID: \"dfc762bd-184c-428f-b6e4-7fd135b57154\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:46:09.417951 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.417893 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvptn\" (UniqueName: \"kubernetes.io/projected/dfc762bd-184c-428f-b6e4-7fd135b57154-kube-api-access-fvptn\") pod \"payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl\" (UID: \"dfc762bd-184c-428f-b6e4-7fd135b57154\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:46:09.418463 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.418443 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc762bd-184c-428f-b6e4-7fd135b57154-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl\" (UID: \"dfc762bd-184c-428f-b6e4-7fd135b57154\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:46:09.474703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.474665 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvptn\" (UniqueName: \"kubernetes.io/projected/dfc762bd-184c-428f-b6e4-7fd135b57154-kube-api-access-fvptn\") pod \"payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl\" (UID: \"dfc762bd-184c-428f-b6e4-7fd135b57154\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:46:09.534459 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.534415 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:46:09.909625 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:09.909600 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl"] Nov 02 06:46:09.910029 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:46:09.909998 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfc762bd_184c_428f_b6e4_7fd135b57154.slice/crio-a1af6c0c96215bf31d5c45ba7ddf2175ce250714a961727730cb1c20dfe1579a WatchSource:0}: Error finding container a1af6c0c96215bf31d5c45ba7ddf2175ce250714a961727730cb1c20dfe1579a: Status 404 returned error can't find the container with id a1af6c0c96215bf31d5c45ba7ddf2175ce250714a961727730cb1c20dfe1579a Nov 02 06:46:10.240263 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.240239 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:46:10.337654 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.337622 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-certs\") pod \"4e49f6b2-4f57-488f-a668-638741a40d60\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " Nov 02 06:46:10.337841 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.337713 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e49f6b2-4f57-488f-a668-638741a40d60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e49f6b2-4f57-488f-a668-638741a40d60-istiod-ca-cert\") pod \"4e49f6b2-4f57-488f-a668-638741a40d60\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " Nov 02 06:46:10.337841 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.337766 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-istio-token\" (UniqueName: \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-istio-token\") pod \"4e49f6b2-4f57-488f-a668-638741a40d60\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " Nov 02 06:46:10.337841 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.337824 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e49f6b2-4f57-488f-a668-638741a40d60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e49f6b2-4f57-488f-a668-638741a40d60-istio-podinfo\") pod \"4e49f6b2-4f57-488f-a668-638741a40d60\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " Nov 02 06:46:10.338019 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.337884 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-data\") pod \"4e49f6b2-4f57-488f-a668-638741a40d60\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " Nov 02 06:46:10.338019 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.337944 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-envoy\") pod \"4e49f6b2-4f57-488f-a668-638741a40d60\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " Nov 02 06:46:10.338158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.337938 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-certs" pod "4e49f6b2-4f57-488f-a668-638741a40d60" (UID: "4e49f6b2-4f57-488f-a668-638741a40d60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.338158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.337975 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-credential-socket\") pod \"4e49f6b2-4f57-488f-a668-638741a40d60\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " Nov 02 06:46:10.338288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.338242 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-data" pod "4e49f6b2-4f57-488f-a668-638741a40d60" (UID: "4e49f6b2-4f57-488f-a668-638741a40d60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.338359 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.338311 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-credential-socket" pod "4e49f6b2-4f57-488f-a668-638741a40d60" (UID: "4e49f6b2-4f57-488f-a668-638741a40d60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.338359 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.338321 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e49f6b2-4f57-488f-a668-638741a40d60-istiod-ca-cert" pod "4e49f6b2-4f57-488f-a668-638741a40d60" (UID: "4e49f6b2-4f57-488f-a668-638741a40d60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:10.338359 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.338315 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-socket\") pod \"4e49f6b2-4f57-488f-a668-638741a40d60\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " Nov 02 06:46:10.338514 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.338442 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-kube-api-access-87ztf\" (UniqueName: \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-kube-api-access-87ztf\") pod \"4e49f6b2-4f57-488f-a668-638741a40d60\" (UID: \"4e49f6b2-4f57-488f-a668-638741a40d60\") " Nov 02 06:46:10.338567 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.338531 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-socket" pod "4e49f6b2-4f57-488f-a668-638741a40d60" (UID: "4e49f6b2-4f57-488f-a668-638741a40d60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.339371 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.339345 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.339483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.339375 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.339483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.339391 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.339483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.339405 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.339483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.339420 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e49f6b2-4f57-488f-a668-638741a40d60-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.340547 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.340522 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e49f6b2-4f57-488f-a668-638741a40d60-istio-podinfo" pod "4e49f6b2-4f57-488f-a668-638741a40d60" (UID: "4e49f6b2-4f57-488f-a668-638741a40d60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:46:10.340693 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.340676 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-istio-token" pod "4e49f6b2-4f57-488f-a668-638741a40d60" (UID: "4e49f6b2-4f57-488f-a668-638741a40d60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:10.340752 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.340733 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-envoy" pod "4e49f6b2-4f57-488f-a668-638741a40d60" (UID: "4e49f6b2-4f57-488f-a668-638741a40d60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:10.340802 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.340783 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-kube-api-access-87ztf" pod "4e49f6b2-4f57-488f-a668-638741a40d60" (UID: "4e49f6b2-4f57-488f-a668-638741a40d60"). InnerVolumeSpecName "kube-api-access-87ztf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:10.441029 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.440983 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e49f6b2-4f57-488f-a668-638741a40d60-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.441029 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.441021 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87ztf\" (UniqueName: \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-kube-api-access-87ztf\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.441029 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.441037 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e49f6b2-4f57-488f-a668-638741a40d60-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.441304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.441049 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e49f6b2-4f57-488f-a668-638741a40d60-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:10.894469 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.894426 5025 generic.go:356] "Generic (PLEG): container finished" podID="4e49f6b2-4f57-488f-a668-638741a40d60" containerID="6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3" exitCode=0 Nov 02 06:46:10.894724 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.894565 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" event={"ID":"4e49f6b2-4f57-488f-a668-638741a40d60","Type":"ContainerDied","Data":"6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3"} Nov 02 06:46:10.894724 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.894621 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" event={"ID":"4e49f6b2-4f57-488f-a668-638741a40d60","Type":"ContainerDied","Data":"81ae8acb0c34e777e922fc790188783787257f27e93f1865faa94f8add632c6c"} Nov 02 06:46:10.894724 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.894653 5025 scope.go:122] "RemoveContainer" containerID="6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3" Nov 02 06:46:10.894918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.894811 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj" Nov 02 06:46:10.897946 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.897914 5025 generic.go:356] "Generic (PLEG): container finished" podID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" containerID="2528049797f3ae3042222247fbd3a3aadbf183d95a86f12cb7e3c4f3af5bb617" exitCode=0 Nov 02 06:46:10.899117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.899067 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" event={"ID":"67dabe99-aa17-4b0d-b582-ef6fbbc826f1","Type":"ContainerDied","Data":"2528049797f3ae3042222247fbd3a3aadbf183d95a86f12cb7e3c4f3af5bb617"} Nov 02 06:46:10.900579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.900543 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" event={"ID":"dfc762bd-184c-428f-b6e4-7fd135b57154","Type":"ContainerStarted","Data":"7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1"} Nov 02 06:46:10.900692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.900596 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" event={"ID":"dfc762bd-184c-428f-b6e4-7fd135b57154","Type":"ContainerStarted","Data":"a1af6c0c96215bf31d5c45ba7ddf2175ce250714a961727730cb1c20dfe1579a"} Nov 02 06:46:10.900737 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.900710 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:46:10.909159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.909142 5025 scope.go:122] "RemoveContainer" containerID="6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3" Nov 02 06:46:10.909450 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:46:10.909420 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3\": container with ID starting with 6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3 not found: ID does not exist" containerID="6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3" Nov 02 06:46:10.909531 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.909461 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3"} err="failed to get container status \"6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3\": rpc error: code = NotFound desc = could not find container \"6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3\": container with ID starting with 6187c30411907d284931d1bb16c21bff7ce32d175f65c6b737d0dd859c5b91c3 not found: ID does not exist" Nov 02 06:46:10.959364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:10.959336 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:46:11.082686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.082610 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj"] Nov 02 06:46:11.129601 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.129565 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wl6mj"] Nov 02 06:46:11.150541 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.150510 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-data\") pod \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " Nov 02 06:46:11.150728 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.150562 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istiod-ca-cert\") pod \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " Nov 02 06:46:11.150796 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.150738 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-kube-api-access-qd5vm\" (UniqueName: \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-kube-api-access-qd5vm\") pod \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " Nov 02 06:46:11.150857 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.150793 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-credential-socket\") pod \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " Nov 02 06:46:11.150918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.150838 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-data" pod "67dabe99-aa17-4b0d-b582-ef6fbbc826f1" (UID: "67dabe99-aa17-4b0d-b582-ef6fbbc826f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.150918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.150876 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-socket\") pod \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " Nov 02 06:46:11.151027 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.150960 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-token\" (UniqueName: \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-token\") pod \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " Nov 02 06:46:11.151027 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.151009 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istiod-ca-cert" pod "67dabe99-aa17-4b0d-b582-ef6fbbc826f1" (UID: "67dabe99-aa17-4b0d-b582-ef6fbbc826f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:11.151150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.151062 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-podinfo\") pod \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " Nov 02 06:46:11.151150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.151113 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-certs\") pod \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " Nov 02 06:46:11.151268 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.151157 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-credential-socket" pod "67dabe99-aa17-4b0d-b582-ef6fbbc826f1" (UID: "67dabe99-aa17-4b0d-b582-ef6fbbc826f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.151268 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.151164 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-envoy\") pod \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\" (UID: \"67dabe99-aa17-4b0d-b582-ef6fbbc826f1\") " Nov 02 06:46:11.151665 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.151634 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-certs" pod "67dabe99-aa17-4b0d-b582-ef6fbbc826f1" (UID: "67dabe99-aa17-4b0d-b582-ef6fbbc826f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.152059 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.152035 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-socket" pod "67dabe99-aa17-4b0d-b582-ef6fbbc826f1" (UID: "67dabe99-aa17-4b0d-b582-ef6fbbc826f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.152233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.152058 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.152350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.152335 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.152464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.152439 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.152464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.152465 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.153486 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.153447 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-envoy" pod "67dabe99-aa17-4b0d-b582-ef6fbbc826f1" (UID: "67dabe99-aa17-4b0d-b582-ef6fbbc826f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:11.153631 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.153613 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-podinfo" pod "67dabe99-aa17-4b0d-b582-ef6fbbc826f1" (UID: "67dabe99-aa17-4b0d-b582-ef6fbbc826f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:46:11.153720 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.153697 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-token" pod "67dabe99-aa17-4b0d-b582-ef6fbbc826f1" (UID: "67dabe99-aa17-4b0d-b582-ef6fbbc826f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:11.153763 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.153736 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-kube-api-access-qd5vm" pod "67dabe99-aa17-4b0d-b582-ef6fbbc826f1" (UID: "67dabe99-aa17-4b0d-b582-ef6fbbc826f1"). InnerVolumeSpecName "kube-api-access-qd5vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:11.191522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.191478 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" podStartSLOduration=2.19146457 podStartE2EDuration="2.19146457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:46:11.190610659 +0000 UTC m=+1747.638987432" watchObservedRunningTime="2026-11-02 06:46:11.19146457 +0000 UTC m=+1747.639841341" Nov 02 06:46:11.253715 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.253686 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd5vm\" (UniqueName: \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-kube-api-access-qd5vm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.253715 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.253712 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.253715 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.253723 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.254386 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.253732 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.254386 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.253741 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67dabe99-aa17-4b0d-b582-ef6fbbc826f1-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:11.670925 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:11.670888 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" path="/var/lib/kubelet/pods/4e49f6b2-4f57-488f-a668-638741a40d60/volumes" Nov 02 06:46:12.166788 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:12.166754 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" Nov 02 06:46:12.166962 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:12.166866 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn" event={"ID":"67dabe99-aa17-4b0d-b582-ef6fbbc826f1","Type":"ContainerDied","Data":"1a2d16e062b20eaf25fa241393ceacbce42a15d00951712c19c69a89eb87a010"} Nov 02 06:46:12.166962 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:12.166917 5025 scope.go:122] "RemoveContainer" containerID="2528049797f3ae3042222247fbd3a3aadbf183d95a86f12cb7e3c4f3af5bb617" Nov 02 06:46:12.375355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:12.375320 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn"] Nov 02 06:46:12.410203 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:12.410167 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-fjlpn"] Nov 02 06:46:12.860103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:12.846051 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:13.671278 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:13.671239 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" path="/var/lib/kubelet/pods/67dabe99-aa17-4b0d-b582-ef6fbbc826f1/volumes" Nov 02 06:46:16.077678 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:16.077639 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:16.782871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:16.782832 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:17.552648 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.552615 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx"] Nov 02 06:46:17.553381 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.553365 5025 cpu_manager.go:414] "containerMap: removing container" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" Nov 02 06:46:17.553445 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.553382 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" Nov 02 06:46:17.553445 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.553398 5025 cpu_manager.go:414] "containerMap: removing container" podUID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" containerName="istio-proxy" Nov 02 06:46:17.553445 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.553404 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" containerName="istio-proxy" Nov 02 06:46:17.553545 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.553510 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e49f6b2-4f57-488f-a668-638741a40d60" containerName="istio-proxy" Nov 02 06:46:17.553545 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.553521 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="67dabe99-aa17-4b0d-b582-ef6fbbc826f1" containerName="istio-proxy" Nov 02 06:46:17.556941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.556923 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.627889 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.627842 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx"] Nov 02 06:46:17.657496 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.657458 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e233a7-openshift-default-dockercfg-5jm84\"" Nov 02 06:46:17.689868 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.689832 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjtrl\" (UniqueName: \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-kube-api-access-kjtrl\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.690039 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.689881 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-data\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.690128 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.690041 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-credential-socket\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.690183 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.690163 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-token\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.690257 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.690242 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-envoy\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.690386 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.690364 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-certs\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.690497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.690484 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istiod-ca-cert\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.690532 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.690516 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-socket\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.690573 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.690546 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-podinfo\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.704281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.704258 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:17.794198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.794150 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-socket\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.794530 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.794500 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-podinfo\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.794782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.794764 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjtrl\" (UniqueName: \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-kube-api-access-kjtrl\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.794899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.794885 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-data\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.795700 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.795669 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-socket\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.795852 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.795831 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-credential-socket\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.795970 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.795220 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-credential-socket\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.796174 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.796155 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-token\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.796397 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.796329 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-envoy\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.803154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.796641 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-certs\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.803551 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.803521 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istiod-ca-cert\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.803694 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.802896 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-certs\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.803768 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.802849 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-data\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.822096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.804981 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istiod-ca-cert\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.822096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.806989 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-podinfo\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.833434 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.833371 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-envoy\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.885165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.882596 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjtrl\" (UniqueName: \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-kube-api-access-kjtrl\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.886299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.886265 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-token\") pod \"e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:17.891412 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:17.887653 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:18.170659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:18.170616 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:18.263418 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:18.262727 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:18.665428 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:46:18.665397 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e66387_4e6c_4d71_9fd5_dd76e420dfd2.slice/crio-49d65cc3382468aa0f04e2ddc097efcec4ee916b8a5bb7b39dc9d99623ae0a99 WatchSource:0}: Error finding container 49d65cc3382468aa0f04e2ddc097efcec4ee916b8a5bb7b39dc9d99623ae0a99: Status 404 returned error can't find the container with id 49d65cc3382468aa0f04e2ddc097efcec4ee916b8a5bb7b39dc9d99623ae0a99 Nov 02 06:46:18.666931 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:18.666908 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx"] Nov 02 06:46:18.668651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:18.668620 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:46:18.668717 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:18.668686 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:46:18.668755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:18.668717 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:46:19.280785 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:19.280749 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" event={"ID":"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2","Type":"ContainerStarted","Data":"58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3"} Nov 02 06:46:19.280965 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:19.280803 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" event={"ID":"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2","Type":"ContainerStarted","Data":"49d65cc3382468aa0f04e2ddc097efcec4ee916b8a5bb7b39dc9d99623ae0a99"} Nov 02 06:46:19.443358 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:19.443299 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" podStartSLOduration=2.443282627 podStartE2EDuration="2.443282627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:46:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:46:19.440460843 +0000 UTC m=+1755.888837622" watchObservedRunningTime="2026-11-02 06:46:19.443282627 +0000 UTC m=+1755.891659383" Nov 02 06:46:20.171170 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:20.171135 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:20.177560 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:20.177531 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:20.287713 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:20.287680 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:20.288653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:20.288632 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:46:22.169876 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:22.169843 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:46:22.178257 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:22.178228 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:22.471777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:22.471697 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:22.707521 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:22.707487 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:23.062442 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:23.060884 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:23.258098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:23.254557 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:23.534365 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:23.534333 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:26.049879 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:26.049846 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:26.293621 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:26.281735 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:28.262439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:28.262406 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:32.602995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:32.602960 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:32.783068 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:46:32.783034 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:46:34.367128 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:34.367093 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:34.627768 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:34.627686 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:34.818282 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:34.818240 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:35.226343 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:35.226313 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:39.903989 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:39.903953 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:41.889100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:41.886596 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:42.078451 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:42.078410 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:42.103399 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:42.103365 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq"] Nov 02 06:46:42.103635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:42.103612 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" containerID="cri-o://27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef" gracePeriod=30 Nov 02 06:46:42.270901 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:42.270870 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:42.458553 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:42.458508 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:43.006099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:43.006054 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:43.197137 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:43.197060 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:44.443809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:44.443762 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:44.943983 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:44.943948 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:45.711376 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:45.711345 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:46.027646 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:46.027567 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:46.963389 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:46.963354 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:47.170993 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.170946 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:47.387332 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.387301 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:46:47.443821 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.443788 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istiod-ca-cert\") pod \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " Nov 02 06:46:47.443998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.443881 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-certs\") pod \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " Nov 02 06:46:47.443998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.443914 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-podinfo\") pod \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " Nov 02 06:46:47.443998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.443963 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-credential-socket\") pod \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " Nov 02 06:46:47.443998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.443991 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-kube-api-access-t9zhl\" (UniqueName: \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-kube-api-access-t9zhl\") pod \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " Nov 02 06:46:47.444202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.444072 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-envoy\") pod \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " Nov 02 06:46:47.444202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.444140 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-socket\") pod \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " Nov 02 06:46:47.444313 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.444238 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-certs" pod "a3992d0d-ee6b-48dc-a41a-dc2302cc0977" (UID: "a3992d0d-ee6b-48dc-a41a-dc2302cc0977"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:47.444313 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.444257 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-data\") pod \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " Nov 02 06:46:47.444313 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.444275 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istiod-ca-cert" pod "a3992d0d-ee6b-48dc-a41a-dc2302cc0977" (UID: "a3992d0d-ee6b-48dc-a41a-dc2302cc0977"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:46:47.444470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.444322 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-credential-socket" pod "a3992d0d-ee6b-48dc-a41a-dc2302cc0977" (UID: "a3992d0d-ee6b-48dc-a41a-dc2302cc0977"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:47.444470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.444388 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-token\" (UniqueName: \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-token\") pod \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\" (UID: \"a3992d0d-ee6b-48dc-a41a-dc2302cc0977\") " Nov 02 06:46:47.444470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.444446 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-data" pod "a3992d0d-ee6b-48dc-a41a-dc2302cc0977" (UID: "a3992d0d-ee6b-48dc-a41a-dc2302cc0977"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:47.444648 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.444575 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-socket" pod "a3992d0d-ee6b-48dc-a41a-dc2302cc0977" (UID: "a3992d0d-ee6b-48dc-a41a-dc2302cc0977"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:47.445289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.445256 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:47.445289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.445286 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:47.445487 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.445303 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:47.445487 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.445319 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:47.445487 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.445334 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:47.446918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.446888 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-kube-api-access-t9zhl" pod "a3992d0d-ee6b-48dc-a41a-dc2302cc0977" (UID: "a3992d0d-ee6b-48dc-a41a-dc2302cc0977"). InnerVolumeSpecName "kube-api-access-t9zhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:47.447030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.446963 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-token" pod "a3992d0d-ee6b-48dc-a41a-dc2302cc0977" (UID: "a3992d0d-ee6b-48dc-a41a-dc2302cc0977"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:46:47.447253 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.447231 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-envoy" pod "a3992d0d-ee6b-48dc-a41a-dc2302cc0977" (UID: "a3992d0d-ee6b-48dc-a41a-dc2302cc0977"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:46:47.447468 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.447441 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-podinfo" pod "a3992d0d-ee6b-48dc-a41a-dc2302cc0977" (UID: "a3992d0d-ee6b-48dc-a41a-dc2302cc0977"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:46:47.528777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.528685 5025 generic.go:356] "Generic (PLEG): container finished" podID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerID="27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef" exitCode=0 Nov 02 06:46:47.528927 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.528780 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" event={"ID":"a3992d0d-ee6b-48dc-a41a-dc2302cc0977","Type":"ContainerDied","Data":"27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef"} Nov 02 06:46:47.528927 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.528803 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" Nov 02 06:46:47.528927 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.528877 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq" event={"ID":"a3992d0d-ee6b-48dc-a41a-dc2302cc0977","Type":"ContainerDied","Data":"08b2319344edf77b9c5d725e987dae0cbdce2e4ca9b90f7f46a49f3530f3cdc6"} Nov 02 06:46:47.528927 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.528905 5025 scope.go:122] "RemoveContainer" containerID="27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef" Nov 02 06:46:47.541555 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.541496 5025 scope.go:122] "RemoveContainer" containerID="36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f" Nov 02 06:46:47.546927 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.546901 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:47.546927 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.546925 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:47.547047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.546936 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9zhl\" (UniqueName: \"kubernetes.io/projected/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-kube-api-access-t9zhl\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:47.547047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.546947 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3992d0d-ee6b-48dc-a41a-dc2302cc0977-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:46:47.552260 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.552225 5025 scope.go:122] "RemoveContainer" containerID="27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef" Nov 02 06:46:47.552535 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:46:47.552517 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef\": container with ID starting with 27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef not found: ID does not exist" containerID="27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef" Nov 02 06:46:47.552592 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.552544 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef"} err="failed to get container status \"27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef\": rpc error: code = NotFound desc = could not find container \"27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef\": container with ID starting with 27946e210a3ce017b7b4d738cefe0d7b8398383d24f836ddc16384ced13441ef not found: ID does not exist" Nov 02 06:46:47.552592 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.552569 5025 scope.go:122] "RemoveContainer" containerID="36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f" Nov 02 06:46:47.552830 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:46:47.552807 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f\": container with ID starting with 36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f not found: ID does not exist" containerID="36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f" Nov 02 06:46:47.552929 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.552833 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f"} err="failed to get container status \"36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f\": rpc error: code = NotFound desc = could not find container \"36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f\": container with ID starting with 36e9198e222491b432d83e0af7b451ad4df78efab18172df399aaec293250d5f not found: ID does not exist" Nov 02 06:46:47.679324 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.679287 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq"] Nov 02 06:46:47.682589 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:47.682560 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e5502564-openshift-default-5bb4587d8f-pttgq"] Nov 02 06:46:49.670687 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:49.670651 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" path="/var/lib/kubelet/pods/a3992d0d-ee6b-48dc-a41a-dc2302cc0977/volumes" Nov 02 06:46:50.205944 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:50.205674 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:53.828499 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:53.828463 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:54.086230 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:54.086146 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:46:55.294106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:46:55.284909 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:00.230590 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:00.230552 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:02.735002 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:47:02.734968 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:47:04.709032 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:04.708991 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:04.941847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:04.941810 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:06.417350 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:06.417310 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh"] Nov 02 06:47:06.417798 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:06.417619 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" containerID="cri-o://b1dd497e535b3b755e1cfcecbabe5d40aee402e1dfa17fb6d2dab5b41596cad5" gracePeriod=30 Nov 02 06:47:09.351101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:09.335031 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:09.534351 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:09.534227 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:10.880095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:10.880041 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:11.794395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.794296 5025 generic.go:356] "Generic (PLEG): container finished" podID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerID="976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4" exitCode=0 Nov 02 06:47:11.794953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.794909 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" event={"ID":"1c0e3851-c4c8-4bed-971c-97f54574135d","Type":"ContainerDied","Data":"976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4"} Nov 02 06:47:11.795782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.795762 5025 scope.go:122] "RemoveContainer" containerID="976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4" Nov 02 06:47:11.802006 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.801786 5025 generic.go:356] "Generic (PLEG): container finished" podID="d090f221-e10d-4266-882a-43e4047e9f33" containerID="b1dd497e535b3b755e1cfcecbabe5d40aee402e1dfa17fb6d2dab5b41596cad5" exitCode=0 Nov 02 06:47:11.802006 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.801892 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" event={"ID":"d090f221-e10d-4266-882a-43e4047e9f33","Type":"ContainerDied","Data":"b1dd497e535b3b755e1cfcecbabe5d40aee402e1dfa17fb6d2dab5b41596cad5"} Nov 02 06:47:11.802006 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.801942 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" event={"ID":"d090f221-e10d-4266-882a-43e4047e9f33","Type":"ContainerDied","Data":"35fc0d9ed27fc8418b1616f2468959c65a38f7a3de37b1000640a0ff9bc8689b"} Nov 02 06:47:11.802006 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.801966 5025 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35fc0d9ed27fc8418b1616f2468959c65a38f7a3de37b1000640a0ff9bc8689b" Nov 02 06:47:11.802507 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.802479 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:47:11.803175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.803142 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:47:11.803294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.803236 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:47:11.803371 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.803320 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:47:11.913056 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913028 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-envoy\") pod \"d090f221-e10d-4266-882a-43e4047e9f33\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " Nov 02 06:47:11.913480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913190 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-data\") pod \"d090f221-e10d-4266-882a-43e4047e9f33\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " Nov 02 06:47:11.913480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913236 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-kube-api-access-99czx\" (UniqueName: \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-kube-api-access-99czx\") pod \"d090f221-e10d-4266-882a-43e4047e9f33\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " Nov 02 06:47:11.913480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913266 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-certs\") pod \"d090f221-e10d-4266-882a-43e4047e9f33\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " Nov 02 06:47:11.913480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913355 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d090f221-e10d-4266-882a-43e4047e9f33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d090f221-e10d-4266-882a-43e4047e9f33-istio-podinfo\") pod \"d090f221-e10d-4266-882a-43e4047e9f33\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " Nov 02 06:47:11.913480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913456 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-istio-token\" (UniqueName: \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-istio-token\") pod \"d090f221-e10d-4266-882a-43e4047e9f33\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " Nov 02 06:47:11.913480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913469 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-data" pod "d090f221-e10d-4266-882a-43e4047e9f33" (UID: "d090f221-e10d-4266-882a-43e4047e9f33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:11.913809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913485 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d090f221-e10d-4266-882a-43e4047e9f33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d090f221-e10d-4266-882a-43e4047e9f33-istiod-ca-cert\") pod \"d090f221-e10d-4266-882a-43e4047e9f33\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " Nov 02 06:47:11.913809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913578 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-credential-socket\") pod \"d090f221-e10d-4266-882a-43e4047e9f33\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " Nov 02 06:47:11.913809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913616 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-certs" pod "d090f221-e10d-4266-882a-43e4047e9f33" (UID: "d090f221-e10d-4266-882a-43e4047e9f33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:11.913809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913642 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-socket\") pod \"d090f221-e10d-4266-882a-43e4047e9f33\" (UID: \"d090f221-e10d-4266-882a-43e4047e9f33\") " Nov 02 06:47:11.914006 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.913928 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d090f221-e10d-4266-882a-43e4047e9f33-istiod-ca-cert" pod "d090f221-e10d-4266-882a-43e4047e9f33" (UID: "d090f221-e10d-4266-882a-43e4047e9f33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:47:11.914355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.914216 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-credential-socket" pod "d090f221-e10d-4266-882a-43e4047e9f33" (UID: "d090f221-e10d-4266-882a-43e4047e9f33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:11.914355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.914304 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-socket" pod "d090f221-e10d-4266-882a-43e4047e9f33" (UID: "d090f221-e10d-4266-882a-43e4047e9f33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:11.914626 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.914601 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.914707 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.914632 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.914707 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.914649 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d090f221-e10d-4266-882a-43e4047e9f33-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.914707 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.914664 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.914707 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.914678 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:11.915387 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.915362 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-envoy" pod "d090f221-e10d-4266-882a-43e4047e9f33" (UID: "d090f221-e10d-4266-882a-43e4047e9f33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:11.915830 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.915802 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-istio-token" pod "d090f221-e10d-4266-882a-43e4047e9f33" (UID: "d090f221-e10d-4266-882a-43e4047e9f33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:11.915905 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.915875 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d090f221-e10d-4266-882a-43e4047e9f33-istio-podinfo" pod "d090f221-e10d-4266-882a-43e4047e9f33" (UID: "d090f221-e10d-4266-882a-43e4047e9f33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:47:11.916894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:11.916878 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-kube-api-access-99czx" pod "d090f221-e10d-4266-882a-43e4047e9f33" (UID: "d090f221-e10d-4266-882a-43e4047e9f33"). InnerVolumeSpecName "kube-api-access-99czx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:12.016847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:12.016695 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.016847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:12.016733 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d090f221-e10d-4266-882a-43e4047e9f33-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.016847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:12.016749 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99czx\" (UniqueName: \"kubernetes.io/projected/d090f221-e10d-4266-882a-43e4047e9f33-kube-api-access-99czx\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.016847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:12.016767 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d090f221-e10d-4266-882a-43e4047e9f33-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:12.810123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:12.810058 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" event={"ID":"1c0e3851-c4c8-4bed-971c-97f54574135d","Type":"ContainerStarted","Data":"c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af"} Nov 02 06:47:12.810361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:12.810099 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh" Nov 02 06:47:12.973468 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:12.973421 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh"] Nov 02 06:47:13.019225 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:13.019189 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-vzhrh"] Nov 02 06:47:13.671461 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:13.670672 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d090f221-e10d-4266-882a-43e4047e9f33" path="/var/lib/kubelet/pods/d090f221-e10d-4266-882a-43e4047e9f33/volumes" Nov 02 06:47:13.750555 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:13.750518 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:47:13.751994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:13.751971 5025 patch_prober.go:29] interesting pod/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 06:47:13.752092 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:13.752020 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 06:47:14.120242 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:14.120112 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:14.750494 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:14.750458 5025 patch_prober.go:29] interesting pod/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 06:47:14.750708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:14.750514 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 06:47:15.455533 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:15.455491 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:15.750743 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:15.750657 5025 patch_prober.go:29] interesting pod/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 06:47:15.750743 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:15.750714 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 06:47:16.750512 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:16.750476 5025 patch_prober.go:29] interesting pod/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 06:47:16.750900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:16.750527 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 06:47:17.711435 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:17.711401 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:47:17.750773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:17.750743 5025 patch_prober.go:29] interesting pod/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" start-of-body= Nov 02 06:47:17.751150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:17.750798 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.112:15021/healthz/ready\": dial tcp 10.133.0.112:15021: connect: connection refused" Nov 02 06:47:18.756151 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:18.756123 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:47:18.757126 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:18.757102 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:47:18.859315 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:18.859280 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:21.904722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:21.904691 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:23.516210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:23.516164 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:23.759544 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:23.759509 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:28.101297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:28.101248 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:32.623331 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:32.623282 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:32.789825 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:47:32.789785 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:47:32.829390 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:32.829350 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:33.121603 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:33.121566 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:37.518962 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:37.518910 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:40.265099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:40.263209 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5"] Nov 02 06:47:40.265099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:40.263476 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" containerID="cri-o://9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0" gracePeriod=30 Nov 02 06:47:40.301121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:40.301055 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl"] Nov 02 06:47:40.301345 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:40.301324 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerName="istio-proxy" containerID="cri-o://652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f" gracePeriod=30 Nov 02 06:47:40.304584 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:40.304558 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g"] Nov 02 06:47:40.304742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:40.304720 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" containerID="cri-o://1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711" gracePeriod=30 Nov 02 06:47:40.348605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:40.348566 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk"] Nov 02 06:47:40.348836 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:40.348812 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerName="istio-proxy" containerID="cri-o://4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e" gracePeriod=30 Nov 02 06:47:42.196633 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:42.196563 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:44.356100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:44.354180 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" podUID="4878a04e-ddb9-4772-a068-3cdba2e83d5c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:47:44.783816 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:44.783761 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:47:44.984239 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:44.984194 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf" podUID="e1b38f9d-ab3b-4722-99b6-98ba6d9972f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 02 06:47:45.088686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.088555 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.52:8000/ready\": context deadline exceeded" Nov 02 06:47:45.682558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.682524 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:47:45.817449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.817423 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:47:45.821555 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.821533 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:47:45.873325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873288 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-podinfo\") pod \"f1abd166-ac13-4582-a584-93ff1f7d8314\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " Nov 02 06:47:45.873505 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873369 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-kube-api-access-78wbh\" (UniqueName: \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-kube-api-access-78wbh\") pod \"b838f9ff-708c-46f5-adec-d4425d46c695\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " Nov 02 06:47:45.873505 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873431 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b838f9ff-708c-46f5-adec-d4425d46c695-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b838f9ff-708c-46f5-adec-d4425d46c695-istiod-ca-cert\") pod \"b838f9ff-708c-46f5-adec-d4425d46c695\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " Nov 02 06:47:45.873505 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873466 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-envoy\") pod \"b838f9ff-708c-46f5-adec-d4425d46c695\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " Nov 02 06:47:45.873742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873508 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0410350-a7d7-46a7-8365-c6ae946c1106-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0410350-a7d7-46a7-8365-c6ae946c1106-istiod-ca-cert\") pod \"d0410350-a7d7-46a7-8365-c6ae946c1106\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " Nov 02 06:47:45.873742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873537 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-certs\") pod \"d0410350-a7d7-46a7-8365-c6ae946c1106\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " Nov 02 06:47:45.873742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873580 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-kube-api-access-lkcpg\" (UniqueName: \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-kube-api-access-lkcpg\") pod \"d0410350-a7d7-46a7-8365-c6ae946c1106\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " Nov 02 06:47:45.873742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873610 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-credential-socket\") pod \"b838f9ff-708c-46f5-adec-d4425d46c695\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " Nov 02 06:47:45.873742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873648 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-socket\") pod \"d0410350-a7d7-46a7-8365-c6ae946c1106\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " Nov 02 06:47:45.873742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873688 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-envoy\") pod \"f1abd166-ac13-4582-a584-93ff1f7d8314\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " Nov 02 06:47:45.874243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873752 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-certs\") pod \"f1abd166-ac13-4582-a584-93ff1f7d8314\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " Nov 02 06:47:45.874243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873826 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-socket\") pod \"f1abd166-ac13-4582-a584-93ff1f7d8314\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " Nov 02 06:47:45.874243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873842 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-certs" pod "d0410350-a7d7-46a7-8365-c6ae946c1106" (UID: "d0410350-a7d7-46a7-8365-c6ae946c1106"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.874243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873871 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-data\") pod \"f1abd166-ac13-4582-a584-93ff1f7d8314\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " Nov 02 06:47:45.874243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873898 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-credential-socket\") pod \"d0410350-a7d7-46a7-8365-c6ae946c1106\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " Nov 02 06:47:45.874243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.873966 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-data\") pod \"d0410350-a7d7-46a7-8365-c6ae946c1106\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " Nov 02 06:47:45.874243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874004 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-certs\") pod \"b838f9ff-708c-46f5-adec-d4425d46c695\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " Nov 02 06:47:45.874243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874055 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-istio-token\" (UniqueName: \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-istio-token\") pod \"b838f9ff-708c-46f5-adec-d4425d46c695\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " Nov 02 06:47:45.874243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874105 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-data\") pod \"b838f9ff-708c-46f5-adec-d4425d46c695\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " Nov 02 06:47:45.874243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874169 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-token\" (UniqueName: \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-token\") pod \"f1abd166-ac13-4582-a584-93ff1f7d8314\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " Nov 02 06:47:45.874243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874227 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-kube-api-access-xgl7j\" (UniqueName: \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-kube-api-access-xgl7j\") pod \"f1abd166-ac13-4582-a584-93ff1f7d8314\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " Nov 02 06:47:45.874817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874298 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-socket\") pod \"b838f9ff-708c-46f5-adec-d4425d46c695\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " Nov 02 06:47:45.874817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874300 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0410350-a7d7-46a7-8365-c6ae946c1106-istiod-ca-cert" pod "d0410350-a7d7-46a7-8365-c6ae946c1106" (UID: "d0410350-a7d7-46a7-8365-c6ae946c1106"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:47:45.874817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874361 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-token\" (UniqueName: \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-token\") pod \"d0410350-a7d7-46a7-8365-c6ae946c1106\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " Nov 02 06:47:45.874817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874366 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b838f9ff-708c-46f5-adec-d4425d46c695-istiod-ca-cert" pod "b838f9ff-708c-46f5-adec-d4425d46c695" (UID: "b838f9ff-708c-46f5-adec-d4425d46c695"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:47:45.874817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874390 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-credential-socket\") pod \"f1abd166-ac13-4582-a584-93ff1f7d8314\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " Nov 02 06:47:45.874817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874430 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-podinfo\") pod \"d0410350-a7d7-46a7-8365-c6ae946c1106\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " Nov 02 06:47:45.874817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874492 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1abd166-ac13-4582-a584-93ff1f7d8314-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1abd166-ac13-4582-a584-93ff1f7d8314-istiod-ca-cert\") pod \"f1abd166-ac13-4582-a584-93ff1f7d8314\" (UID: \"f1abd166-ac13-4582-a584-93ff1f7d8314\") " Nov 02 06:47:45.874817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874517 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-data" pod "f1abd166-ac13-4582-a584-93ff1f7d8314" (UID: "f1abd166-ac13-4582-a584-93ff1f7d8314"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.874817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874534 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-envoy\") pod \"d0410350-a7d7-46a7-8365-c6ae946c1106\" (UID: \"d0410350-a7d7-46a7-8365-c6ae946c1106\") " Nov 02 06:47:45.874817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874586 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b838f9ff-708c-46f5-adec-d4425d46c695-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b838f9ff-708c-46f5-adec-d4425d46c695-istio-podinfo\") pod \"b838f9ff-708c-46f5-adec-d4425d46c695\" (UID: \"b838f9ff-708c-46f5-adec-d4425d46c695\") " Nov 02 06:47:45.874817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874628 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-data" pod "b838f9ff-708c-46f5-adec-d4425d46c695" (UID: "b838f9ff-708c-46f5-adec-d4425d46c695"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.875388 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.874842 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-credential-socket" pod "d0410350-a7d7-46a7-8365-c6ae946c1106" (UID: "d0410350-a7d7-46a7-8365-c6ae946c1106"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.875388 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.875191 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-data" pod "d0410350-a7d7-46a7-8365-c6ae946c1106" (UID: "d0410350-a7d7-46a7-8365-c6ae946c1106"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.876135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.875463 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.876135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.875492 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.876135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.875509 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.876135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.875523 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.876135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.875537 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b838f9ff-708c-46f5-adec-d4425d46c695-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.876135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.875550 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0410350-a7d7-46a7-8365-c6ae946c1106-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.876135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.875565 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.876135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.875863 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-certs" pod "b838f9ff-708c-46f5-adec-d4425d46c695" (UID: "b838f9ff-708c-46f5-adec-d4425d46c695"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.877510 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.877466 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b838f9ff-708c-46f5-adec-d4425d46c695-istio-podinfo" pod "b838f9ff-708c-46f5-adec-d4425d46c695" (UID: "b838f9ff-708c-46f5-adec-d4425d46c695"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:47:45.877990 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.877711 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-credential-socket" pod "f1abd166-ac13-4582-a584-93ff1f7d8314" (UID: "f1abd166-ac13-4582-a584-93ff1f7d8314"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.877990 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.877828 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-envoy" pod "f1abd166-ac13-4582-a584-93ff1f7d8314" (UID: "f1abd166-ac13-4582-a584-93ff1f7d8314"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.878315 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.878210 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-credential-socket" pod "b838f9ff-708c-46f5-adec-d4425d46c695" (UID: "b838f9ff-708c-46f5-adec-d4425d46c695"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.878778 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.878529 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-socket" pod "d0410350-a7d7-46a7-8365-c6ae946c1106" (UID: "d0410350-a7d7-46a7-8365-c6ae946c1106"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.910144 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.900630 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-istio-token" pod "b838f9ff-708c-46f5-adec-d4425d46c695" (UID: "b838f9ff-708c-46f5-adec-d4425d46c695"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:45.910683 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.910466 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-token" pod "d0410350-a7d7-46a7-8365-c6ae946c1106" (UID: "d0410350-a7d7-46a7-8365-c6ae946c1106"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:45.911005 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.910839 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-socket" pod "b838f9ff-708c-46f5-adec-d4425d46c695" (UID: "b838f9ff-708c-46f5-adec-d4425d46c695"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.914972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.911158 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-certs" pod "f1abd166-ac13-4582-a584-93ff1f7d8314" (UID: "f1abd166-ac13-4582-a584-93ff1f7d8314"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.914972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.911502 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1abd166-ac13-4582-a584-93ff1f7d8314-istiod-ca-cert" pod "f1abd166-ac13-4582-a584-93ff1f7d8314" (UID: "f1abd166-ac13-4582-a584-93ff1f7d8314"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:47:45.914972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.912154 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-podinfo" pod "f1abd166-ac13-4582-a584-93ff1f7d8314" (UID: "f1abd166-ac13-4582-a584-93ff1f7d8314"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:47:45.914972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.913117 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-envoy" pod "b838f9ff-708c-46f5-adec-d4425d46c695" (UID: "b838f9ff-708c-46f5-adec-d4425d46c695"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.914972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.913141 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-podinfo" pod "d0410350-a7d7-46a7-8365-c6ae946c1106" (UID: "d0410350-a7d7-46a7-8365-c6ae946c1106"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:47:45.914972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.913228 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-kube-api-access-78wbh" pod "b838f9ff-708c-46f5-adec-d4425d46c695" (UID: "b838f9ff-708c-46f5-adec-d4425d46c695"). InnerVolumeSpecName "kube-api-access-78wbh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:45.914972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.914667 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-socket" pod "f1abd166-ac13-4582-a584-93ff1f7d8314" (UID: "f1abd166-ac13-4582-a584-93ff1f7d8314"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.914972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.914764 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-kube-api-access-xgl7j" pod "f1abd166-ac13-4582-a584-93ff1f7d8314" (UID: "f1abd166-ac13-4582-a584-93ff1f7d8314"). InnerVolumeSpecName "kube-api-access-xgl7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:45.914972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.914855 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-kube-api-access-lkcpg" pod "d0410350-a7d7-46a7-8365-c6ae946c1106" (UID: "d0410350-a7d7-46a7-8365-c6ae946c1106"). InnerVolumeSpecName "kube-api-access-lkcpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:45.914972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.914940 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-envoy" pod "d0410350-a7d7-46a7-8365-c6ae946c1106" (UID: "d0410350-a7d7-46a7-8365-c6ae946c1106"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:45.915443 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.914980 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-token" pod "f1abd166-ac13-4582-a584-93ff1f7d8314" (UID: "f1abd166-ac13-4582-a584-93ff1f7d8314"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:45.978917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.978804 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.978917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.978853 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkcpg\" (UniqueName: \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-kube-api-access-lkcpg\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.978917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.978871 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.978917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.978886 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.979462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.979386 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.979462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.979414 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.979899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.979444 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.979899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.979639 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.979899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.979653 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.979899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.979666 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.979899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.979679 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgl7j\" (UniqueName: \"kubernetes.io/projected/f1abd166-ac13-4582-a584-93ff1f7d8314-kube-api-access-xgl7j\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.979899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.979692 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b838f9ff-708c-46f5-adec-d4425d46c695-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.980423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.980290 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.980423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.980317 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1abd166-ac13-4582-a584-93ff1f7d8314-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.980423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.980333 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.980423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.980347 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1abd166-ac13-4582-a584-93ff1f7d8314-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.980423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.980361 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0410350-a7d7-46a7-8365-c6ae946c1106-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.980423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.980377 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b838f9ff-708c-46f5-adec-d4425d46c695-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.980423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.980390 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1abd166-ac13-4582-a584-93ff1f7d8314-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:45.980423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:45.980404 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78wbh\" (UniqueName: \"kubernetes.io/projected/b838f9ff-708c-46f5-adec-d4425d46c695-kube-api-access-78wbh\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.053298 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.053263 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:47:46.125101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.124224 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-podinfo\") pod \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " Nov 02 06:47:46.125101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.124296 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-envoy\") pod \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " Nov 02 06:47:46.125101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.124487 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istiod-ca-cert\") pod \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " Nov 02 06:47:46.125101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.124518 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-kube-api-access-fhjtc\" (UniqueName: \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-kube-api-access-fhjtc\") pod \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " Nov 02 06:47:46.125101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.124561 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-data\") pod \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " Nov 02 06:47:46.125101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.124642 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-token\" (UniqueName: \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-token\") pod \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " Nov 02 06:47:46.125101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.124677 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-socket\") pod \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " Nov 02 06:47:46.125101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.124720 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-certs\") pod \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " Nov 02 06:47:46.125101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.124827 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-credential-socket\") pod \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\" (UID: \"b6ca03e3-b80b-498c-af13-b549f3e1ab54\") " Nov 02 06:47:46.127044 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.126844 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istiod-ca-cert" pod "b6ca03e3-b80b-498c-af13-b549f3e1ab54" (UID: "b6ca03e3-b80b-498c-af13-b549f3e1ab54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:47:46.127899 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.127870 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-kube-api-access-fhjtc" pod "b6ca03e3-b80b-498c-af13-b549f3e1ab54" (UID: "b6ca03e3-b80b-498c-af13-b549f3e1ab54"). InnerVolumeSpecName "kube-api-access-fhjtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:46.128426 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.128385 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-podinfo" pod "b6ca03e3-b80b-498c-af13-b549f3e1ab54" (UID: "b6ca03e3-b80b-498c-af13-b549f3e1ab54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:47:46.129775 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.129667 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-data" pod "b6ca03e3-b80b-498c-af13-b549f3e1ab54" (UID: "b6ca03e3-b80b-498c-af13-b549f3e1ab54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:46.129775 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.129736 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-socket" pod "b6ca03e3-b80b-498c-af13-b549f3e1ab54" (UID: "b6ca03e3-b80b-498c-af13-b549f3e1ab54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:46.129775 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.129753 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-credential-socket" pod "b6ca03e3-b80b-498c-af13-b549f3e1ab54" (UID: "b6ca03e3-b80b-498c-af13-b549f3e1ab54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:46.130067 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.129814 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-token" pod "b6ca03e3-b80b-498c-af13-b549f3e1ab54" (UID: "b6ca03e3-b80b-498c-af13-b549f3e1ab54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:47:46.130067 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.129924 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-envoy" pod "b6ca03e3-b80b-498c-af13-b549f3e1ab54" (UID: "b6ca03e3-b80b-498c-af13-b549f3e1ab54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:46.130067 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.129942 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-certs" pod "b6ca03e3-b80b-498c-af13-b549f3e1ab54" (UID: "b6ca03e3-b80b-498c-af13-b549f3e1ab54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:47:46.150977 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.150952 5025 generic.go:356] "Generic (PLEG): container finished" podID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerID="4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e" exitCode=0 Nov 02 06:47:46.151107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.151007 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" Nov 02 06:47:46.151107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.151035 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" event={"ID":"d0410350-a7d7-46a7-8365-c6ae946c1106","Type":"ContainerDied","Data":"4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e"} Nov 02 06:47:46.151233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.151139 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk" event={"ID":"d0410350-a7d7-46a7-8365-c6ae946c1106","Type":"ContainerDied","Data":"e95755de3675382e4fce973936001f9ed06ef82a86cdb8ad9c56d0fc78877dd5"} Nov 02 06:47:46.151233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.151167 5025 scope.go:122] "RemoveContainer" containerID="4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e" Nov 02 06:47:46.152558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.152528 5025 generic.go:356] "Generic (PLEG): container finished" podID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerID="652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f" exitCode=0 Nov 02 06:47:46.152643 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.152598 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" event={"ID":"f1abd166-ac13-4582-a584-93ff1f7d8314","Type":"ContainerDied","Data":"652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f"} Nov 02 06:47:46.152708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.152654 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" event={"ID":"f1abd166-ac13-4582-a584-93ff1f7d8314","Type":"ContainerDied","Data":"d3b4efd39cf959f1ade3a4d249ba6cdb64bb8dac6949ef2336a9fa364a9bc616"} Nov 02 06:47:46.152708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.152606 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl" Nov 02 06:47:46.154226 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.154199 5025 generic.go:356] "Generic (PLEG): container finished" podID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerID="1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711" exitCode=0 Nov 02 06:47:46.154318 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.154266 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" event={"ID":"b6ca03e3-b80b-498c-af13-b549f3e1ab54","Type":"ContainerDied","Data":"1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711"} Nov 02 06:47:46.154318 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.154295 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" event={"ID":"b6ca03e3-b80b-498c-af13-b549f3e1ab54","Type":"ContainerDied","Data":"b97d7a824f0f2cee7555f3ced1e70318af4276642ec1138a8acf5dfa9c07a59c"} Nov 02 06:47:46.154433 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.154351 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g" Nov 02 06:47:46.161809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.161746 5025 generic.go:356] "Generic (PLEG): container finished" podID="b838f9ff-708c-46f5-adec-d4425d46c695" containerID="9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0" exitCode=0 Nov 02 06:47:46.161886 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.161830 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" event={"ID":"b838f9ff-708c-46f5-adec-d4425d46c695","Type":"ContainerDied","Data":"9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0"} Nov 02 06:47:46.161886 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.161868 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" event={"ID":"b838f9ff-708c-46f5-adec-d4425d46c695","Type":"ContainerDied","Data":"ec1362123961f517895c0f89255caac5591d72a4d8c94cf80db1449f9c8ec3b7"} Nov 02 06:47:46.161955 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.161931 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5" Nov 02 06:47:46.163613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.163597 5025 scope.go:122] "RemoveContainer" containerID="4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e" Nov 02 06:47:46.163925 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:47:46.163908 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e\": container with ID starting with 4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e not found: ID does not exist" containerID="4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e" Nov 02 06:47:46.163964 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.163936 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e"} err="failed to get container status \"4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e\": rpc error: code = NotFound desc = could not find container \"4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e\": container with ID starting with 4b3f09650c7b764fc83cf8b9460d84d8acc3702cece08abcb851137c4bd0f64e not found: ID does not exist" Nov 02 06:47:46.163964 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.163958 5025 scope.go:122] "RemoveContainer" containerID="652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f" Nov 02 06:47:46.177867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.176258 5025 scope.go:122] "RemoveContainer" containerID="652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f" Nov 02 06:47:46.177867 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:47:46.176486 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f\": container with ID starting with 652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f not found: ID does not exist" containerID="652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f" Nov 02 06:47:46.177867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.176508 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f"} err="failed to get container status \"652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f\": rpc error: code = NotFound desc = could not find container \"652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f\": container with ID starting with 652ea5f5dedb327c7475ea52b0a4195b34d200c4870ee9d3f9a1708c800c099f not found: ID does not exist" Nov 02 06:47:46.177867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.176523 5025 scope.go:122] "RemoveContainer" containerID="1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711" Nov 02 06:47:46.211270 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.211249 5025 scope.go:122] "RemoveContainer" containerID="1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711" Nov 02 06:47:46.211583 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:47:46.211556 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711\": container with ID starting with 1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711 not found: ID does not exist" containerID="1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711" Nov 02 06:47:46.211692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.211590 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711"} err="failed to get container status \"1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711\": rpc error: code = NotFound desc = could not find container \"1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711\": container with ID starting with 1bc0d6cab9a05bdae891a76420c586daf9cafced63fe32af147fb24736d72711 not found: ID does not exist" Nov 02 06:47:46.211692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.211612 5025 scope.go:122] "RemoveContainer" containerID="9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0" Nov 02 06:47:46.224762 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.224691 5025 scope.go:122] "RemoveContainer" containerID="9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0" Nov 02 06:47:46.225270 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:47:46.225246 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0\": container with ID starting with 9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0 not found: ID does not exist" containerID="9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0" Nov 02 06:47:46.225426 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.225405 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0"} err="failed to get container status \"9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0\": rpc error: code = NotFound desc = could not find container \"9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0\": container with ID starting with 9e75cd73e20b9b3a51b23552251d8221ba7606f335ed1e6cfbccc55d886fe2a0 not found: ID does not exist" Nov 02 06:47:46.227687 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.227652 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.227840 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.227810 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.227994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.227946 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.227994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.227967 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.228222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.228127 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhjtc\" (UniqueName: \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-kube-api-access-fhjtc\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.228222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.228166 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.228222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.228180 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6ca03e3-b80b-498c-af13-b549f3e1ab54-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.228222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.228191 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.228222 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.228199 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6ca03e3-b80b-498c-af13-b549f3e1ab54-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:47:46.331099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.331057 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl"] Nov 02 06:47:46.335988 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.335959 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-wrkhl"] Nov 02 06:47:46.351388 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.351361 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:46.508100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.501719 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g"] Nov 02 06:47:46.526098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.523988 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-pjv7g"] Nov 02 06:47:46.618165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.618131 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk"] Nov 02 06:47:46.642104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.640947 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ksftk"] Nov 02 06:47:46.722210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.722135 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5"] Nov 02 06:47:46.778205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:46.775820 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-tlwz5"] Nov 02 06:47:47.672368 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:47.672324 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" path="/var/lib/kubelet/pods/b6ca03e3-b80b-498c-af13-b549f3e1ab54/volumes" Nov 02 06:47:47.672974 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:47.672950 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" path="/var/lib/kubelet/pods/b838f9ff-708c-46f5-adec-d4425d46c695/volumes" Nov 02 06:47:47.673550 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:47.673528 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" path="/var/lib/kubelet/pods/d0410350-a7d7-46a7-8365-c6ae946c1106/volumes" Nov 02 06:47:47.674412 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:47.674391 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" path="/var/lib/kubelet/pods/f1abd166-ac13-4582-a584-93ff1f7d8314/volumes" Nov 02 06:47:48.109211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:48.109110 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:48.320097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:48.308038 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:52.299900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.299407 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:52.791892 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.791860 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8"] Nov 02 06:47:52.792668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792647 5025 cpu_manager.go:414] "containerMap: removing container" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" Nov 02 06:47:52.792668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792668 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792690 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792697 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792719 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792726 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792737 5025 cpu_manager.go:414] "containerMap: removing container" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792744 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792763 5025 cpu_manager.go:414] "containerMap: removing container" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792770 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792798 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792806 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792835 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" Nov 02 06:47:52.792858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792843 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" Nov 02 06:47:52.793484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.792995 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" Nov 02 06:47:52.793484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.793010 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1abd166-ac13-4582-a584-93ff1f7d8314" containerName="istio-proxy" Nov 02 06:47:52.793484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.793028 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b838f9ff-708c-46f5-adec-d4425d46c695" containerName="istio-proxy" Nov 02 06:47:52.793484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.793042 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ca03e3-b80b-498c-af13-b549f3e1ab54" containerName="istio-proxy" Nov 02 06:47:52.793484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.793054 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0410350-a7d7-46a7-8365-c6ae946c1106" containerName="istio-proxy" Nov 02 06:47:52.793484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.793066 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d090f221-e10d-4266-882a-43e4047e9f33" containerName="istio-proxy" Nov 02 06:47:52.796746 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.796725 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:52.850622 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.850581 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8"] Nov 02 06:47:52.893002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.892972 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e233a7-serving-cert\"" Nov 02 06:47:52.901810 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.901772 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85be5a1-c949-4a43-9c96-a236c69e1b6c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:52.901920 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.901829 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-metrics-certs\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:52.901997 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.901970 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-maas-api-tls\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:52.902167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:52.902153 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdmv8\" (UniqueName: \"kubernetes.io/projected/b85be5a1-c949-4a43-9c96-a236c69e1b6c-kube-api-access-rdmv8\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:53.003735 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.003701 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdmv8\" (UniqueName: \"kubernetes.io/projected/b85be5a1-c949-4a43-9c96-a236c69e1b6c-kube-api-access-rdmv8\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:53.003930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.003808 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85be5a1-c949-4a43-9c96-a236c69e1b6c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:53.003930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.003838 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-metrics-certs\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:53.003930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.003896 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-maas-api-tls\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:53.004054 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:47:53.004016 5025 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-serving-cert: secret "maas-api-e2e-api-b-e233a7-serving-cert" not found Nov 02 06:47:53.004110 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:47:53.004071 5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-maas-api-tls podName:b85be5a1-c949-4a43-9c96-a236c69e1b6c nodeName:}" failed. No retries permitted until 2026-11-02 06:47:53.504056547 +0000 UTC m=+1849.952433294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-maas-api-tls") pod "maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" (UID: "b85be5a1-c949-4a43-9c96-a236c69e1b6c") : secret "maas-api-e2e-api-b-e233a7-serving-cert" not found Nov 02 06:47:53.004458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.004437 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85be5a1-c949-4a43-9c96-a236c69e1b6c-openshift-service-ca\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:53.006342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.006320 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-metrics-certs\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:53.037120 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.036753 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:53.077560 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.077476 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdmv8\" (UniqueName: \"kubernetes.io/projected/b85be5a1-c949-4a43-9c96-a236c69e1b6c-kube-api-access-rdmv8\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:53.232136 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.231668 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:53.256062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.256029 5025 generic.go:356] "Generic (PLEG): container finished" podID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerID="56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f" exitCode=0 Nov 02 06:47:53.256298 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.256104 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" event={"ID":"bc2f04e7-114a-4ab0-8e5f-1035460391fd","Type":"ContainerDied","Data":"56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f"} Nov 02 06:47:53.256618 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.256600 5025 scope.go:122] "RemoveContainer" containerID="56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f" Nov 02 06:47:53.258474 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.258441 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:47:53.258555 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.258513 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:47:53.258600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.258552 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:47:53.420677 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.420642 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:53.514786 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.514756 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-maas-api-tls\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:53.517156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.517135 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-maas-api-tls\") pod \"maas-api-e2e-api-b-e233a7-756f895c6c-m29s8\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:53.710996 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:53.710967 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:54.110468 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:54.110431 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8"] Nov 02 06:47:54.112612 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:47:54.112563 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85be5a1_c949_4a43_9c96_a236c69e1b6c.slice/crio-cbeeb8cddb89f5f64edb140002de4a30c2e178db1467a9018a7dce31ceb0a531 WatchSource:0}: Error finding container cbeeb8cddb89f5f64edb140002de4a30c2e178db1467a9018a7dce31ceb0a531: Status 404 returned error can't find the container with id cbeeb8cddb89f5f64edb140002de4a30c2e178db1467a9018a7dce31ceb0a531 Nov 02 06:47:54.263492 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:54.263453 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" event={"ID":"b85be5a1-c949-4a43-9c96-a236c69e1b6c","Type":"ContainerStarted","Data":"cbeeb8cddb89f5f64edb140002de4a30c2e178db1467a9018a7dce31ceb0a531"} Nov 02 06:47:54.265442 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:54.265414 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" event={"ID":"bc2f04e7-114a-4ab0-8e5f-1035460391fd","Type":"ContainerStarted","Data":"517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53"} Nov 02 06:47:55.021583 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:55.021510 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:47:55.023110 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:55.022974 5025 patch_prober.go:29] interesting pod/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 06:47:55.023110 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:55.023045 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 06:47:55.273981 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:55.273878 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" event={"ID":"b85be5a1-c949-4a43-9c96-a236c69e1b6c","Type":"ContainerStarted","Data":"9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e"} Nov 02 06:47:55.273981 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:55.273961 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:47:55.714912 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:55.714876 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:55.935736 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:55.935685 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:56.025024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:56.023478 5025 patch_prober.go:29] interesting pod/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 06:47:56.025024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:56.023540 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 06:47:57.021877 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:57.021841 5025 patch_prober.go:29] interesting pod/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 06:47:57.022051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:57.021892 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 06:47:58.021481 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.021450 5025 patch_prober.go:29] interesting pod/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 06:47:58.021894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.021501 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 06:47:58.199430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.199395 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:58.257198 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.257164 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:47:58.405155 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.401943 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:58.646111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.618922 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:58.752235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.752182 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" podStartSLOduration=5.843208688 podStartE2EDuration="6.752168471s" totalImagesPullingTime="908.959783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:47:55.402317294 +0000 UTC m=+1851.850694075" watchObservedRunningTime="2026-11-02 06:47:58.752168471 +0000 UTC m=+1855.200545275" Nov 02 06:47:58.754098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.754062 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn"] Nov 02 06:47:58.754822 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.754806 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3992d0d-ee6b-48dc-a41a-dc2302cc0977" containerName="istio-proxy" Nov 02 06:47:58.758425 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.758406 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:47:58.864100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.864005 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3bf02ebf-dockercfg-z69gv\"" Nov 02 06:47:58.868098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.864289 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3bf02ebf\"" Nov 02 06:47:58.868098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.865995 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7gph\" (UniqueName: \"kubernetes.io/projected/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-kube-api-access-m7gph\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn\" (UID: \"aa8bbf32-36d7-497f-8deb-c8afebfe67d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:47:58.868098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.866044 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn\" (UID: \"aa8bbf32-36d7-497f-8deb-c8afebfe67d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:47:58.875432 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:58.875399 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn"] Nov 02 06:47:59.059161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.001999 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7gph\" (UniqueName: \"kubernetes.io/projected/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-kube-api-access-m7gph\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn\" (UID: \"aa8bbf32-36d7-497f-8deb-c8afebfe67d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:47:59.059161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.002101 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn\" (UID: \"aa8bbf32-36d7-497f-8deb-c8afebfe67d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:47:59.059161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.003260 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn\" (UID: \"aa8bbf32-36d7-497f-8deb-c8afebfe67d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:47:59.127940 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.089525 5025 patch_prober.go:29] interesting pod/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body= Nov 02 06:47:59.127940 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.089612 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" Nov 02 06:47:59.132669 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.132613 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj"] Nov 02 06:47:59.139431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.139307 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj"] Nov 02 06:47:59.139653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.139405 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:47:59.144129 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.144103 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7gph\" (UniqueName: \"kubernetes.io/projected/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-kube-api-access-m7gph\") pod \"payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn\" (UID: \"aa8bbf32-36d7-497f-8deb-c8afebfe67d4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:47:59.228464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.228432 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8dc6\" (UniqueName: \"kubernetes.io/projected/60030f97-89f9-4b64-980f-787c1e4c832a-kube-api-access-m8dc6\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj\" (UID: \"60030f97-89f9-4b64-980f-787c1e4c832a\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:47:59.228757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.228739 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60030f97-89f9-4b64-980f-787c1e4c832a-plugins-config-volume\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj\" (UID: \"60030f97-89f9-4b64-980f-787c1e4c832a\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:47:59.330536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.330442 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8dc6\" (UniqueName: \"kubernetes.io/projected/60030f97-89f9-4b64-980f-787c1e4c832a-kube-api-access-m8dc6\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj\" (UID: \"60030f97-89f9-4b64-980f-787c1e4c832a\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:47:59.330536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.330514 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60030f97-89f9-4b64-980f-787c1e4c832a-plugins-config-volume\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj\" (UID: \"60030f97-89f9-4b64-980f-787c1e4c832a\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:47:59.331229 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.331209 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60030f97-89f9-4b64-980f-787c1e4c832a-plugins-config-volume\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj\" (UID: \"60030f97-89f9-4b64-980f-787c1e4c832a\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:47:59.381517 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.381487 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:47:59.391607 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.391573 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8dc6\" (UniqueName: \"kubernetes.io/projected/60030f97-89f9-4b64-980f-787c1e4c832a-kube-api-access-m8dc6\") pod \"payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj\" (UID: \"60030f97-89f9-4b64-980f-787c1e4c832a\") " pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:47:59.416351 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.416313 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:47:59.468478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.468445 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:47:59.760245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.760220 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn"] Nov 02 06:47:59.760831 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:47:59.760809 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8bbf32_36d7_497f_8deb_c8afebfe67d4.slice/crio-34dd4b648c012416a2d4c9d0a9c0238a3e3141c4abd22c69b1320b92b1f2bfa7 WatchSource:0}: Error finding container 34dd4b648c012416a2d4c9d0a9c0238a3e3141c4abd22c69b1320b92b1f2bfa7: Status 404 returned error can't find the container with id 34dd4b648c012416a2d4c9d0a9c0238a3e3141c4abd22c69b1320b92b1f2bfa7 Nov 02 06:47:59.854364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:47:59.854337 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj"] Nov 02 06:47:59.854991 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:47:59.854967 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60030f97_89f9_4b64_980f_787c1e4c832a.slice/crio-961f54c3b0651d05e0386408768c482be0b6d0985cdb82da28e51558ca11a48a WatchSource:0}: Error finding container 961f54c3b0651d05e0386408768c482be0b6d0985cdb82da28e51558ca11a48a: Status 404 returned error can't find the container with id 961f54c3b0651d05e0386408768c482be0b6d0985cdb82da28e51558ca11a48a Nov 02 06:48:00.027234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:00.027164 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:48:00.028483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:00.028460 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:48:00.336680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:00.336591 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" event={"ID":"60030f97-89f9-4b64-980f-787c1e4c832a","Type":"ContainerStarted","Data":"f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c"} Nov 02 06:48:00.336680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:00.336656 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" event={"ID":"60030f97-89f9-4b64-980f-787c1e4c832a","Type":"ContainerStarted","Data":"961f54c3b0651d05e0386408768c482be0b6d0985cdb82da28e51558ca11a48a"} Nov 02 06:48:00.336680 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:00.336680 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:48:00.339004 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:00.338976 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" event={"ID":"aa8bbf32-36d7-497f-8deb-c8afebfe67d4","Type":"ContainerStarted","Data":"718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8"} Nov 02 06:48:00.339130 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:00.339021 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:48:00.339130 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:00.339036 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" event={"ID":"aa8bbf32-36d7-497f-8deb-c8afebfe67d4","Type":"ContainerStarted","Data":"34dd4b648c012416a2d4c9d0a9c0238a3e3141c4abd22c69b1320b92b1f2bfa7"} Nov 02 06:48:00.452562 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:00.452501 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" podStartSLOduration=2.452483925 podStartE2EDuration="2.452483925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:48:00.451456762 +0000 UTC m=+1856.899833518" watchObservedRunningTime="2026-11-02 06:48:00.452483925 +0000 UTC m=+1856.900860682" Nov 02 06:48:00.573109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:00.571874 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" podStartSLOduration=2.571755876 podStartE2EDuration="2.571755876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:48:00.563253254 +0000 UTC m=+1857.011630009" watchObservedRunningTime="2026-11-02 06:48:00.571755876 +0000 UTC m=+1857.020132648" Nov 02 06:48:01.287938 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:01.287912 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:48:02.708123 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:48:02.708087 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:48:03.255814 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:03.255725 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:04.283259 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:04.283229 5025 scope.go:122] "RemoveContainer" containerID="b1dd497e535b3b755e1cfcecbabe5d40aee402e1dfa17fb6d2dab5b41596cad5" Nov 02 06:48:08.415703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:08.415640 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:10.482876 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.482840 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt"] Nov 02 06:48:10.513569 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.513536 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.530674 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.530647 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx"] Nov 02 06:48:10.541650 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.541627 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh"] Nov 02 06:48:10.541804 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.541774 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.547057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.547037 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.574050 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.574029 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-hptbf\"" Nov 02 06:48:10.579883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.579861 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt"] Nov 02 06:48:10.620645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.620614 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx"] Nov 02 06:48:10.627919 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.627881 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh"] Nov 02 06:48:10.654968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.654933 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.655264 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.655238 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.655421 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.655402 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.655547 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.655531 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.655704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.655685 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w562\" (UniqueName: \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-kube-api-access-7w562\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.655850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.655834 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.655974 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.655958 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.656110 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.656094 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.656244 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.656228 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnh26\" (UniqueName: \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-kube-api-access-dnh26\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.656377 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.656361 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.656510 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.656494 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp4l8\" (UniqueName: \"kubernetes.io/projected/c1cf776b-6c17-4791-9a7b-927b1bc044c0-kube-api-access-kp4l8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.656678 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.656660 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.656797 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.656781 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.656916 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.656901 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.657106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.657055 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.657283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.657263 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.657470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.657452 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.657601 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.657585 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.657750 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.657728 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.657890 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.657875 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.658014 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.657999 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.658159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.658141 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.658279 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.658265 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.658438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.658422 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.658569 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.658553 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.658708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.658691 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.658826 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.658811 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.760763 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.760685 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.760763 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.760722 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.760763 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.760747 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.761161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.760789 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.761161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.760813 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.761284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761171 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.761284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761206 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.761284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761243 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.761456 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761394 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w562\" (UniqueName: \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-kube-api-access-7w562\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.761456 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761445 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.761584 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761477 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.761584 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761510 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.761584 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761546 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnh26\" (UniqueName: \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-kube-api-access-dnh26\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.761742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761600 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.761742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761638 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp4l8\" (UniqueName: \"kubernetes.io/projected/c1cf776b-6c17-4791-9a7b-927b1bc044c0-kube-api-access-kp4l8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.761742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761696 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.761742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761727 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.762057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761760 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.762057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761839 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.762057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761900 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.762057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.761962 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.762057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762000 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.762057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762014 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.762057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762025 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.762439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762092 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.762439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762142 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.762439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762180 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.762439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762220 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.762439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762238 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.762439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762300 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.762439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762311 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.762439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762394 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.762439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762400 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.762878 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762484 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.762878 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762799 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.762878 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.762852 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.763037 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.763002 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.763187 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.763145 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.763705 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.763673 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.764178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.763747 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.764178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.764024 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.764372 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.764347 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.764519 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.764501 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.764854 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.764796 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.764974 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.764964 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.765554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.765529 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.765886 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.765865 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.766632 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.766615 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.816570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.816065 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp4l8\" (UniqueName: \"kubernetes.io/projected/c1cf776b-6c17-4791-9a7b-927b1bc044c0-kube-api-access-kp4l8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.817602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.817541 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnh26\" (UniqueName: \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-kube-api-access-dnh26\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.818052 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.818034 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1cf776b-6c17-4791-9a7b-927b1bc044c0-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-4t2wh\" (UID: \"c1cf776b-6c17-4791-9a7b-927b1bc044c0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:10.818449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.818431 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.818558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.818537 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w562\" (UniqueName: \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-kube-api-access-7w562\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-ct8dx\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.818751 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.818736 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nrmmt\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.828501 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.828483 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:10.858141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.858119 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:10.864430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:10.864407 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:11.279483 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.279455 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt"] Nov 02 06:48:11.279870 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:48:11.279848 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec594c99_c66a_4ca2_b126_db0cbb8e9a1c.slice/crio-f2c9b7c30b6ea9ef88b63f325f91a3a13de03a7eaf1c9ccdff9fcaf04b6e1eec WatchSource:0}: Error finding container f2c9b7c30b6ea9ef88b63f325f91a3a13de03a7eaf1c9ccdff9fcaf04b6e1eec: Status 404 returned error can't find the container with id f2c9b7c30b6ea9ef88b63f325f91a3a13de03a7eaf1c9ccdff9fcaf04b6e1eec Nov 02 06:48:11.282953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.282918 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:11.283052 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.282985 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:11.283052 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.283014 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:11.310130 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.283557 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx"] Nov 02 06:48:11.317635 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:48:11.317605 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3951ca8a_a28d_410d_bfad_a5ea1e49fd51.slice/crio-6166a0e8b6c4b43f961794e4ccc47090422597abd6fe2f7c150ec07d28fd4d1d WatchSource:0}: Error finding container 6166a0e8b6c4b43f961794e4ccc47090422597abd6fe2f7c150ec07d28fd4d1d: Status 404 returned error can't find the container with id 6166a0e8b6c4b43f961794e4ccc47090422597abd6fe2f7c150ec07d28fd4d1d Nov 02 06:48:11.320641 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.320597 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:11.320728 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.320701 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:11.320768 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.320753 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:11.332123 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:48:11.332095 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cf776b_6c17_4791_9a7b_927b1bc044c0.slice/crio-06298629ace800d7b03e452aed9f767ebe087d0d70d4c927721de1f984cf96d2 WatchSource:0}: Error finding container 06298629ace800d7b03e452aed9f767ebe087d0d70d4c927721de1f984cf96d2: Status 404 returned error can't find the container with id 06298629ace800d7b03e452aed9f767ebe087d0d70d4c927721de1f984cf96d2 Nov 02 06:48:11.332822 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.332798 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh"] Nov 02 06:48:11.334629 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.334601 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:11.334695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.334663 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:11.334695 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.334692 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:11.348874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.348849 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:48:11.349316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.349296 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:48:11.650128 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.638304 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" event={"ID":"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c","Type":"ContainerStarted","Data":"09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2"} Nov 02 06:48:11.650128 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.638509 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" event={"ID":"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c","Type":"ContainerStarted","Data":"f2c9b7c30b6ea9ef88b63f325f91a3a13de03a7eaf1c9ccdff9fcaf04b6e1eec"} Nov 02 06:48:11.650128 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.648482 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" event={"ID":"c1cf776b-6c17-4791-9a7b-927b1bc044c0","Type":"ContainerStarted","Data":"43043e526a6bb1c0c5f32569fb31a82c1d6eb0c52d6802bef8a0d5135d583856"} Nov 02 06:48:11.650128 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.648566 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" event={"ID":"c1cf776b-6c17-4791-9a7b-927b1bc044c0","Type":"ContainerStarted","Data":"06298629ace800d7b03e452aed9f767ebe087d0d70d4c927721de1f984cf96d2"} Nov 02 06:48:11.652653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.652528 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" event={"ID":"3951ca8a-a28d-410d-bfad-a5ea1e49fd51","Type":"ContainerStarted","Data":"b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da"} Nov 02 06:48:11.652953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.652867 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" event={"ID":"3951ca8a-a28d-410d-bfad-a5ea1e49fd51","Type":"ContainerStarted","Data":"6166a0e8b6c4b43f961794e4ccc47090422597abd6fe2f7c150ec07d28fd4d1d"} Nov 02 06:48:11.827322 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.827252 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" podStartSLOduration=1.827232252 podStartE2EDuration="1.827232252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:48:11.81020329 +0000 UTC m=+1868.258580059" watchObservedRunningTime="2026-11-02 06:48:11.827232252 +0000 UTC m=+1868.275609008" Nov 02 06:48:11.829290 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.829266 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:11.858704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.858672 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:11.865442 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.865416 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:11.932816 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:11.932751 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" podStartSLOduration=1.932732315 podStartE2EDuration="1.932732315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:48:11.924404509 +0000 UTC m=+1868.372781266" watchObservedRunningTime="2026-11-02 06:48:11.932732315 +0000 UTC m=+1868.381109078" Nov 02 06:48:12.037008 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:12.036935 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podStartSLOduration=2.036915785 podStartE2EDuration="2.036915785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:48:12.035716799 +0000 UTC m=+1868.484093555" watchObservedRunningTime="2026-11-02 06:48:12.036915785 +0000 UTC m=+1868.485292541" Nov 02 06:48:12.890402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:12.890358 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 06:48:12.907232 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:12.907180 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 06:48:13.175211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.149748 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:48:13.175211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.149820 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 06:48:13.176310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.175563 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:48:13.176310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.175628 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 06:48:13.462236 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.434742 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:13.812152 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.808936 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": context deadline exceeded" Nov 02 06:48:13.849100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.839677 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:48:13.849100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.839747 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 06:48:13.861472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.860204 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 06:48:13.861472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.860284 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 06:48:13.878461 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.878426 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:48:13.882150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:13.882114 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 06:48:14.835102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:14.831445 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:48:14.835102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:14.831514 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 06:48:14.866520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:14.866476 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 06:48:14.866731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:14.866539 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 06:48:14.866731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:14.866480 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:48:14.866731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:14.866602 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 06:48:15.852738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:15.852221 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:48:15.852738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:15.852304 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 06:48:15.860623 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:15.860477 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 06:48:15.860623 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:15.860545 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 06:48:15.864868 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:15.864823 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:48:15.865015 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:15.864899 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 06:48:16.834101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:16.833227 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:48:16.834101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:16.833307 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 06:48:16.860757 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:16.860722 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 06:48:16.861400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:16.861340 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 06:48:16.866287 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:16.866257 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:48:16.866438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:16.866322 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 06:48:17.846098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:17.837766 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:48:17.846098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:17.837865 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 06:48:17.862976 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:17.861868 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 06:48:17.862976 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:17.861925 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 06:48:17.865768 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:17.865725 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:48:17.865939 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:17.865921 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 06:48:18.100157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:18.096490 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:18.315229 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:18.315179 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:18.525999 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:18.525919 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:18.829948 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:18.829865 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:48:18.829948 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:18.829936 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 06:48:18.859159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:18.859118 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 06:48:18.859306 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:18.859186 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 06:48:18.864983 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:18.864935 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:48:18.865440 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:18.865013 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 06:48:19.359891 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:19.359784 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:19.562161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:19.549006 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:19.675233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:19.675202 5025 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:48:19.675419 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:19.675263 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:48:19.829611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:19.829576 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Nov 02 06:48:19.829820 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:19.829642 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Nov 02 06:48:19.859638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:19.859472 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" start-of-body= Nov 02 06:48:19.859638 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:19.859529 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.127:15021/healthz/ready\": dial tcp 10.133.0.127:15021: connect: connection refused" Nov 02 06:48:19.865431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:19.865401 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" start-of-body= Nov 02 06:48:19.865861 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:19.865453 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": dial tcp 10.133.0.128:15021: connect: connection refused" Nov 02 06:48:21.817102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:21.814338 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:21.817102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:21.815730 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:21.817102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:21.817063 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 06:48:21.871524 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:21.871210 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:48:21.871524 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:21.871262 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:48:21.908223 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:21.908192 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:48:21.923043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:21.908277 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:48:21.990973 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:21.990926 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:21.992874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:21.991491 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:21.992874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:21.992471 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 06:48:22.014577 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:22.011760 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:22.014577 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:22.012313 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:22.014577 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:22.013533 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" Nov 02 06:48:22.964429 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:22.964397 5025 generic.go:356] "Generic (PLEG): container finished" podID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerID="356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8" exitCode=0 Nov 02 06:48:22.964856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:22.964467 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" event={"ID":"c68a4b47-40b0-493f-a07e-721871e27a0f","Type":"ContainerDied","Data":"356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8"} Nov 02 06:48:22.965425 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:22.965401 5025 scope.go:122] "RemoveContainer" containerID="356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8" Nov 02 06:48:22.966865 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:22.966837 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:22.966920 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:22.966902 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:22.966956 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:22.966935 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:23.474000 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:23.473959 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:23.973466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:23.973431 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" event={"ID":"c68a4b47-40b0-493f-a07e-721871e27a0f","Type":"ContainerStarted","Data":"a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407"} Nov 02 06:48:24.103100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:24.093266 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:24.965647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:24.965607 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:48:24.965837 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:24.965816 5025 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 02 06:48:24.965885 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:24.965866 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 02 06:48:25.966325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:25.966289 5025 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 02 06:48:25.966875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:25.966352 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 02 06:48:26.010608 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:26.010571 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:26.208947 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:26.208906 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:26.966390 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:26.966357 5025 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 02 06:48:26.966773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:26.966407 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 02 06:48:27.966235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:27.966199 5025 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 02 06:48:27.966396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:27.966265 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 02 06:48:28.204455 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:28.204416 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:28.966277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:28.966244 5025 patch_prober.go:29] interesting pod/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body= Nov 02 06:48:28.966453 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:28.966309 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" Nov 02 06:48:29.970591 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:29.970563 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:48:29.971071 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:29.970902 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:48:29.971515 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:29.971494 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:48:32.708238 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:48:32.708205 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:48:32.895946 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:32.895853 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:36.466900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:36.466865 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf"] Nov 02 06:48:36.467340 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:36.467069 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" containerID="cri-o://5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911" gracePeriod=30 Nov 02 06:48:37.522300 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:37.522254 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:40.026141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:40.026091 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:40.215651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:40.215624 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:41.786987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.786957 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:48:41.893210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.893171 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-podinfo\") pod \"ab885c6e-6f00-421e-8ca1-313e33e7efba\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " Nov 02 06:48:41.893403 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.893221 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-kube-api-access-zjth7\" (UniqueName: \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-kube-api-access-zjth7\") pod \"ab885c6e-6f00-421e-8ca1-313e33e7efba\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " Nov 02 06:48:41.893403 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.893280 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-socket\") pod \"ab885c6e-6f00-421e-8ca1-313e33e7efba\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " Nov 02 06:48:41.893403 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.893326 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-envoy\") pod \"ab885c6e-6f00-421e-8ca1-313e33e7efba\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " Nov 02 06:48:41.893593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.893529 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-credential-socket\") pod \"ab885c6e-6f00-421e-8ca1-313e33e7efba\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " Nov 02 06:48:41.893788 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.893756 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-socket" pod "ab885c6e-6f00-421e-8ca1-313e33e7efba" (UID: "ab885c6e-6f00-421e-8ca1-313e33e7efba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:41.893972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.893840 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-certs\") pod \"ab885c6e-6f00-421e-8ca1-313e33e7efba\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " Nov 02 06:48:41.894062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.893769 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-credential-socket" pod "ab885c6e-6f00-421e-8ca1-313e33e7efba" (UID: "ab885c6e-6f00-421e-8ca1-313e33e7efba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:41.894062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.894017 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-certs" pod "ab885c6e-6f00-421e-8ca1-313e33e7efba" (UID: "ab885c6e-6f00-421e-8ca1-313e33e7efba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:41.895539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.894127 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab885c6e-6f00-421e-8ca1-313e33e7efba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab885c6e-6f00-421e-8ca1-313e33e7efba-istiod-ca-cert\") pod \"ab885c6e-6f00-421e-8ca1-313e33e7efba\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " Nov 02 06:48:41.895539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.894193 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-token\" (UniqueName: \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-token\") pod \"ab885c6e-6f00-421e-8ca1-313e33e7efba\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " Nov 02 06:48:41.895539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.894252 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-data\") pod \"ab885c6e-6f00-421e-8ca1-313e33e7efba\" (UID: \"ab885c6e-6f00-421e-8ca1-313e33e7efba\") " Nov 02 06:48:41.895539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.894860 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab885c6e-6f00-421e-8ca1-313e33e7efba-istiod-ca-cert" pod "ab885c6e-6f00-421e-8ca1-313e33e7efba" (UID: "ab885c6e-6f00-421e-8ca1-313e33e7efba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:48:41.895539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.895092 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-data" pod "ab885c6e-6f00-421e-8ca1-313e33e7efba" (UID: "ab885c6e-6f00-421e-8ca1-313e33e7efba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:41.895539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.895252 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.895539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.895292 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.895539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.895308 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab885c6e-6f00-421e-8ca1-313e33e7efba-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.896113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.896060 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-kube-api-access-zjth7" pod "ab885c6e-6f00-421e-8ca1-313e33e7efba" (UID: "ab885c6e-6f00-421e-8ca1-313e33e7efba"). InnerVolumeSpecName "kube-api-access-zjth7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:41.896244 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.896191 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-podinfo" pod "ab885c6e-6f00-421e-8ca1-313e33e7efba" (UID: "ab885c6e-6f00-421e-8ca1-313e33e7efba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:48:41.896330 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.896244 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-envoy" pod "ab885c6e-6f00-421e-8ca1-313e33e7efba" (UID: "ab885c6e-6f00-421e-8ca1-313e33e7efba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:48:41.896941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.896922 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-token" pod "ab885c6e-6f00-421e-8ca1-313e33e7efba" (UID: "ab885c6e-6f00-421e-8ca1-313e33e7efba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:48:41.997343 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.997305 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.997343 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.997346 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.997595 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.997362 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.997595 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.997407 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjth7\" (UniqueName: \"kubernetes.io/projected/ab885c6e-6f00-421e-8ca1-313e33e7efba-kube-api-access-zjth7\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.997595 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.997424 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:41.997595 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:41.997438 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab885c6e-6f00-421e-8ca1-313e33e7efba-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:48:42.125856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:42.125765 5025 generic.go:356] "Generic (PLEG): container finished" podID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerID="5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911" exitCode=0 Nov 02 06:48:42.125856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:42.125826 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" Nov 02 06:48:42.126104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:42.125845 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" event={"ID":"ab885c6e-6f00-421e-8ca1-313e33e7efba","Type":"ContainerDied","Data":"5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911"} Nov 02 06:48:42.126104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:42.125899 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf" event={"ID":"ab885c6e-6f00-421e-8ca1-313e33e7efba","Type":"ContainerDied","Data":"6ed74f23456112f15fa3db1660630dd7bfc5450f436f04c882cb7b396d8ebaff"} Nov 02 06:48:42.126104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:42.125912 5025 scope.go:122] "RemoveContainer" containerID="5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911" Nov 02 06:48:42.135184 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:42.135158 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:42.136895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:42.136880 5025 scope.go:122] "RemoveContainer" containerID="5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911" Nov 02 06:48:42.137179 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:48:42.137161 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911\": container with ID starting with 5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911 not found: ID does not exist" containerID="5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911" Nov 02 06:48:42.137238 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:42.137188 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911"} err="failed to get container status \"5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911\": rpc error: code = NotFound desc = could not find container \"5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911\": container with ID starting with 5b52e99c0912278c7535ed8e28346a8f672d0243a58c1ee4004106086d998911 not found: ID does not exist" Nov 02 06:48:42.309208 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:42.309152 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf"] Nov 02 06:48:42.309926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:42.309902 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-snnhf"] Nov 02 06:48:43.670663 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:43.670630 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" path="/var/lib/kubelet/pods/ab885c6e-6f00-421e-8ca1-313e33e7efba/volumes" Nov 02 06:48:45.813694 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:45.813661 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:46.271056 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.271020 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:46.628773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.628689 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8"] Nov 02 06:48:46.629441 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.629427 5025 cpu_manager.go:414] "containerMap: removing container" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" Nov 02 06:48:46.629504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.629442 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" Nov 02 06:48:46.629546 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.629537 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab885c6e-6f00-421e-8ca1-313e33e7efba" containerName="istio-proxy" Nov 02 06:48:46.634601 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.634584 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:46.729552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.729525 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a2ba11-openshift-default-dockercfg-bgfj9\"" Nov 02 06:48:46.735957 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.735927 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8"] Nov 02 06:48:46.760204 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.760159 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:46.807105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.788578 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istiod-ca-cert\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:46.816825 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.816789 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-certs\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:46.827878 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.827717 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-token\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:46.835396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.835360 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-envoy\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:46.871167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.842432 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-podinfo\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:46.871739 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.871574 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthlw\" (UniqueName: \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-kube-api-access-gthlw\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:46.872058 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.872039 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-data\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:46.874494 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.874472 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-socket\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:46.875912 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:46.875889 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-credential-socket\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.112889 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.112853 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:47.116991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.116962 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-credential-socket\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.117148 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.117058 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istiod-ca-cert\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.117148 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.117104 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-certs\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.117221 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.117146 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-token\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.117221 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.117198 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-envoy\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.117288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.117277 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-podinfo\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.117346 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.117332 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthlw\" (UniqueName: \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-kube-api-access-gthlw\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.117429 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.117404 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-credential-socket\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.118069 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.118043 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-certs\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.118897 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.117387 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-data\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.119269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.119231 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-socket\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.120669 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.120554 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istiod-ca-cert\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.124605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.124579 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-data\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.124716 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.124704 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-socket\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.126110 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.125237 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-podinfo\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.126401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.126360 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-envoy\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.175205 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.175174 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-token\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.175417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.175395 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthlw\" (UniqueName: \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-kube-api-access-gthlw\") pod \"e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.249838 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.249802 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:47.304546 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.304507 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:47.710717 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:48:47.710676 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90a63809_f2cc_4157_80e4_d5cba8ddb07d.slice/crio-c3b2b7eb7a7e13d65e2b6dad40a35ad57911ca98d0416a3a90e1f2926c79b827 WatchSource:0}: Error finding container c3b2b7eb7a7e13d65e2b6dad40a35ad57911ca98d0416a3a90e1f2926c79b827: Status 404 returned error can't find the container with id c3b2b7eb7a7e13d65e2b6dad40a35ad57911ca98d0416a3a90e1f2926c79b827 Nov 02 06:48:47.713844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.713797 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:47.713966 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.713893 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:47.713966 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.713936 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:47.715361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:47.715341 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8"] Nov 02 06:48:48.258739 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:48.258685 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" event={"ID":"90a63809-f2cc-4157-80e4-d5cba8ddb07d","Type":"ContainerStarted","Data":"896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe"} Nov 02 06:48:48.259273 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:48.258766 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" event={"ID":"90a63809-f2cc-4157-80e4-d5cba8ddb07d","Type":"ContainerStarted","Data":"c3b2b7eb7a7e13d65e2b6dad40a35ad57911ca98d0416a3a90e1f2926c79b827"} Nov 02 06:48:48.467892 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:48.467837 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" podStartSLOduration=2.467822836 podStartE2EDuration="2.467822836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:48:48.417412876 +0000 UTC m=+1904.865789630" watchObservedRunningTime="2026-11-02 06:48:48.467822836 +0000 UTC m=+1904.916199602" Nov 02 06:48:49.250145 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:49.250064 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:49.256093 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:49.256057 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:49.273481 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:49.273447 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:49.274826 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:49.274804 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:48:51.212061 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:51.212022 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:51.445917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:51.445880 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:52.234097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:52.230807 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:53.918428 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:53.918390 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:55.273808 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:55.273777 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:55.486881 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:55.486845 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:57.091480 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:57.091372 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:48:57.501874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:57.501843 5025 generic.go:356] "Generic (PLEG): container finished" podID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerID="58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3" exitCode=0 Nov 02 06:48:57.502056 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:57.501913 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" event={"ID":"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2","Type":"ContainerDied","Data":"58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3"} Nov 02 06:48:57.502459 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:57.502443 5025 scope.go:122] "RemoveContainer" containerID="58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3" Nov 02 06:48:57.504310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:57.504282 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:57.504390 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:57.504350 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:57.504390 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:57.504381 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:48:58.516691 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:58.516658 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" event={"ID":"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2","Type":"ContainerStarted","Data":"7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f"} Nov 02 06:48:59.171231 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:59.171198 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:48:59.171422 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:59.171401 5025 patch_prober.go:29] interesting pod/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 06:48:59.171481 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:48:59.171461 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 06:49:00.171119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:00.171056 5025 patch_prober.go:29] interesting pod/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 06:49:00.171502 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:00.171144 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 06:49:00.849749 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:00.849714 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq"] Nov 02 06:49:00.947928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:00.947893 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg"] Nov 02 06:49:01.171811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:01.171777 5025 patch_prober.go:29] interesting pod/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 06:49:01.172204 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:01.171831 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 06:49:01.257353 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:01.257322 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:01.457735 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:01.457648 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:01.736144 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:01.736030 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:01.914951 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:01.914926 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:02.171955 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:02.171922 5025 patch_prober.go:29] interesting pod/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 06:49:02.172442 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:02.171987 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 06:49:02.735055 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:49:02.735015 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:49:03.171711 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:03.171675 5025 patch_prober.go:29] interesting pod/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Nov 02 06:49:03.171879 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:03.171730 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Nov 02 06:49:04.175558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:04.175528 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:49:04.175965 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:04.175943 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:49:04.176717 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:04.176698 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:49:05.850900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:05.850854 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" podUID="99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb" containerName="payload-processing" containerID="cri-o://7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91" gracePeriod=25 Nov 02 06:49:05.948414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:05.948375 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" podUID="6aee84db-e7cf-41f6-9e44-e981dacfb8cc" containerName="payload-pre-processing" containerID="cri-o://58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57" gracePeriod=25 Nov 02 06:49:06.222251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.222226 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:49:06.266275 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.266253 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:49:06.361219 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.361188 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-kube-api-access-zm9kj\" (UniqueName: \"kubernetes.io/projected/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-kube-api-access-zm9kj\") pod \"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb\" (UID: \"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb\") " Nov 02 06:49:06.361612 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.361585 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-kube-api-access-dhjkg\" (UniqueName: \"kubernetes.io/projected/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-kube-api-access-dhjkg\") pod \"6aee84db-e7cf-41f6-9e44-e981dacfb8cc\" (UID: \"6aee84db-e7cf-41f6-9e44-e981dacfb8cc\") " Nov 02 06:49:06.361857 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.361831 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-plugins-config-volume\") pod \"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb\" (UID: \"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb\") " Nov 02 06:49:06.361962 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.361883 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-plugins-config-volume\") pod \"6aee84db-e7cf-41f6-9e44-e981dacfb8cc\" (UID: \"6aee84db-e7cf-41f6-9e44-e981dacfb8cc\") " Nov 02 06:49:06.362297 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.362257 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-plugins-config-volume" pod "99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb" (UID: "99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:49:06.362500 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.362478 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-plugins-config-volume" pod "6aee84db-e7cf-41f6-9e44-e981dacfb8cc" (UID: "6aee84db-e7cf-41f6-9e44-e981dacfb8cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:49:06.362650 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.362629 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:06.362717 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.362655 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:06.422659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.420731 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:06.425331 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.425260 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-kube-api-access-dhjkg" pod "6aee84db-e7cf-41f6-9e44-e981dacfb8cc" (UID: "6aee84db-e7cf-41f6-9e44-e981dacfb8cc"). InnerVolumeSpecName "kube-api-access-dhjkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:06.437053 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.436792 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-kube-api-access-zm9kj" pod "99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb" (UID: "99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb"). InnerVolumeSpecName "kube-api-access-zm9kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:06.464281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.464251 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhjkg\" (UniqueName: \"kubernetes.io/projected/6aee84db-e7cf-41f6-9e44-e981dacfb8cc-kube-api-access-dhjkg\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:06.464281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.464279 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm9kj\" (UniqueName: \"kubernetes.io/projected/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb-kube-api-access-zm9kj\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:06.632059 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.632022 5025 generic.go:356] "Generic (PLEG): container finished" podID="6aee84db-e7cf-41f6-9e44-e981dacfb8cc" containerID="58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57" exitCode=0 Nov 02 06:49:06.632059 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.632054 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" Nov 02 06:49:06.632310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.632065 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" event={"ID":"6aee84db-e7cf-41f6-9e44-e981dacfb8cc","Type":"ContainerDied","Data":"58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57"} Nov 02 06:49:06.632310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.632132 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg" event={"ID":"6aee84db-e7cf-41f6-9e44-e981dacfb8cc","Type":"ContainerDied","Data":"836f1ecd1b6e8f7a76ebfc4a56af0523fb55efbafbd6a7f75d350334107c464d"} Nov 02 06:49:06.632310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.632155 5025 scope.go:122] "RemoveContainer" containerID="58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57" Nov 02 06:49:06.633620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.633596 5025 generic.go:356] "Generic (PLEG): container finished" podID="99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb" containerID="7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91" exitCode=0 Nov 02 06:49:06.633758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.633647 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" Nov 02 06:49:06.633758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.633676 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" event={"ID":"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb","Type":"ContainerDied","Data":"7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91"} Nov 02 06:49:06.633758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.633719 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq" event={"ID":"99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb","Type":"ContainerDied","Data":"9ff3464c5ea4e3592e75d566a138dd407ffdcb392b4233b7f44ffe5f85105856"} Nov 02 06:49:06.642860 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.642842 5025 scope.go:122] "RemoveContainer" containerID="58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57" Nov 02 06:49:06.643204 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:49:06.643179 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57\": container with ID starting with 58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57 not found: ID does not exist" containerID="58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57" Nov 02 06:49:06.643311 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.643211 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57"} err="failed to get container status \"58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57\": rpc error: code = NotFound desc = could not find container \"58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57\": container with ID starting with 58900171f0a71d6ed9e1d9fb91db69670afd8cf22a15857e124c3ef6aceccd57 not found: ID does not exist" Nov 02 06:49:06.643311 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.643227 5025 scope.go:122] "RemoveContainer" containerID="7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91" Nov 02 06:49:06.652197 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.652182 5025 scope.go:122] "RemoveContainer" containerID="7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91" Nov 02 06:49:06.652440 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:49:06.652420 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91\": container with ID starting with 7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91 not found: ID does not exist" containerID="7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91" Nov 02 06:49:06.652490 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.652446 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91"} err="failed to get container status \"7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91\": rpc error: code = NotFound desc = could not find container \"7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91\": container with ID starting with 7f90ee9fe7cdabb429dca33a0c9624d68ec3945e3d1241c060a0b75e661c6c91 not found: ID does not exist" Nov 02 06:49:06.790933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.790842 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg"] Nov 02 06:49:06.794063 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.794039 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8ecb23-7769786bf5-mfkhg"] Nov 02 06:49:06.898504 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.898466 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq"] Nov 02 06:49:06.901581 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:06.901558 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8ecb23-b9b5fbd5c-lt7kq"] Nov 02 06:49:07.670133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:07.670101 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aee84db-e7cf-41f6-9e44-e981dacfb8cc" path="/var/lib/kubelet/pods/6aee84db-e7cf-41f6-9e44-e981dacfb8cc/volumes" Nov 02 06:49:07.670579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:07.670560 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb" path="/var/lib/kubelet/pods/99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb/volumes" Nov 02 06:49:11.245112 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:11.243427 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:16.420282 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:16.420241 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:21.089235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:21.089196 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:21.303965 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:21.303927 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:21.525998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:21.525951 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:24.453948 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:24.453916 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:26.514613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:26.514563 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:31.495101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:31.477582 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:32.707626 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:49:32.707590 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:49:35.841760 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:35.841709 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:37.634937 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:37.634896 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:37.826335 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:37.826286 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:37.915132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:37.915070 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr"] Nov 02 06:49:38.005889 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:38.005857 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:38.206655 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:38.206565 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:38.394565 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:38.394527 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:38.590856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:38.590765 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:42.662220 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:42.661496 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:42.916017 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:42.915978 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" podUID="8bd5a095-dd31-4987-bf49-4fd9616a022c" containerName="payload-pre-processing" containerID="cri-o://17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c" gracePeriod=25 Nov 02 06:49:43.189392 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.189363 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:49:43.261608 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.261575 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd5a095-dd31-4987-bf49-4fd9616a022c-kube-api-access-zt668\" (UniqueName: \"kubernetes.io/projected/8bd5a095-dd31-4987-bf49-4fd9616a022c-kube-api-access-zt668\") pod \"8bd5a095-dd31-4987-bf49-4fd9616a022c\" (UID: \"8bd5a095-dd31-4987-bf49-4fd9616a022c\") " Nov 02 06:49:43.261779 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.261731 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd5a095-dd31-4987-bf49-4fd9616a022c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd5a095-dd31-4987-bf49-4fd9616a022c-plugins-config-volume\") pod \"8bd5a095-dd31-4987-bf49-4fd9616a022c\" (UID: \"8bd5a095-dd31-4987-bf49-4fd9616a022c\") " Nov 02 06:49:43.262132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.262107 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd5a095-dd31-4987-bf49-4fd9616a022c-plugins-config-volume" pod "8bd5a095-dd31-4987-bf49-4fd9616a022c" (UID: "8bd5a095-dd31-4987-bf49-4fd9616a022c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:49:43.263615 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.263596 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd5a095-dd31-4987-bf49-4fd9616a022c-kube-api-access-zt668" pod "8bd5a095-dd31-4987-bf49-4fd9616a022c" (UID: "8bd5a095-dd31-4987-bf49-4fd9616a022c"). InnerVolumeSpecName "kube-api-access-zt668". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:43.363152 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.363120 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt668\" (UniqueName: \"kubernetes.io/projected/8bd5a095-dd31-4987-bf49-4fd9616a022c-kube-api-access-zt668\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:43.363152 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.363148 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bd5a095-dd31-4987-bf49-4fd9616a022c-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:43.951491 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.951393 5025 generic.go:356] "Generic (PLEG): container finished" podID="8bd5a095-dd31-4987-bf49-4fd9616a022c" containerID="17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c" exitCode=0 Nov 02 06:49:43.951491 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.951464 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" Nov 02 06:49:43.951960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.951484 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" event={"ID":"8bd5a095-dd31-4987-bf49-4fd9616a022c","Type":"ContainerDied","Data":"17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c"} Nov 02 06:49:43.951960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.951570 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr" event={"ID":"8bd5a095-dd31-4987-bf49-4fd9616a022c","Type":"ContainerDied","Data":"7bd5381b1ae7fe17a8a95aae4ce30351ddcdc3da713d4d2491ed193a1503b85b"} Nov 02 06:49:43.951960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.951596 5025 scope.go:122] "RemoveContainer" containerID="17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c" Nov 02 06:49:43.962176 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.962161 5025 scope.go:122] "RemoveContainer" containerID="17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c" Nov 02 06:49:43.962460 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:49:43.962440 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c\": container with ID starting with 17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c not found: ID does not exist" containerID="17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c" Nov 02 06:49:43.962562 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:43.962465 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c"} err="failed to get container status \"17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c\": rpc error: code = NotFound desc = could not find container \"17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c\": container with ID starting with 17f25f97c63c69ae10bc5d9a0b2a6d7cc33e6573ebf2c04d43409a16b447891c not found: ID does not exist" Nov 02 06:49:44.109891 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:44.109857 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr"] Nov 02 06:49:44.156679 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:44.156648 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-70898f-6f79c77b79-6mprr"] Nov 02 06:49:44.951302 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:44.951273 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:45.129527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:45.129499 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:45.217948 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:45.217877 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg"] Nov 02 06:49:45.218124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:45.218098 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" containerID="cri-o://c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af" gracePeriod=30 Nov 02 06:49:45.324099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:45.320327 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:45.669527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:45.669494 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd5a095-dd31-4987-bf49-4fd9616a022c" path="/var/lib/kubelet/pods/8bd5a095-dd31-4987-bf49-4fd9616a022c/volumes" Nov 02 06:49:47.283103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:47.281530 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:48.758994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:48.758954 5025 patch_prober.go:29] interesting pod/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:49:48.759410 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:48.759024 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:49:50.504365 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.504339 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:49:50.584034 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584008 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-certs\") pod \"1c0e3851-c4c8-4bed-971c-97f54574135d\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " Nov 02 06:49:50.584214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584055 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-kube-api-access-6zrpz\" (UniqueName: \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-kube-api-access-6zrpz\") pod \"1c0e3851-c4c8-4bed-971c-97f54574135d\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " Nov 02 06:49:50.584214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584111 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-credential-socket\") pod \"1c0e3851-c4c8-4bed-971c-97f54574135d\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " Nov 02 06:49:50.584214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584192 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-socket\") pod \"1c0e3851-c4c8-4bed-971c-97f54574135d\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " Nov 02 06:49:50.584385 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584236 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-token\" (UniqueName: \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-token\") pod \"1c0e3851-c4c8-4bed-971c-97f54574135d\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " Nov 02 06:49:50.584385 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584325 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-certs" pod "1c0e3851-c4c8-4bed-971c-97f54574135d" (UID: "1c0e3851-c4c8-4bed-971c-97f54574135d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:49:50.584385 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584338 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-envoy\") pod \"1c0e3851-c4c8-4bed-971c-97f54574135d\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " Nov 02 06:49:50.584515 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584413 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-data\") pod \"1c0e3851-c4c8-4bed-971c-97f54574135d\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " Nov 02 06:49:50.584515 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584465 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-socket" pod "1c0e3851-c4c8-4bed-971c-97f54574135d" (UID: "1c0e3851-c4c8-4bed-971c-97f54574135d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:49:50.584515 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584475 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-credential-socket" pod "1c0e3851-c4c8-4bed-971c-97f54574135d" (UID: "1c0e3851-c4c8-4bed-971c-97f54574135d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:49:50.584683 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584567 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-podinfo\") pod \"1c0e3851-c4c8-4bed-971c-97f54574135d\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " Nov 02 06:49:50.584683 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584619 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0e3851-c4c8-4bed-971c-97f54574135d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0e3851-c4c8-4bed-971c-97f54574135d-istiod-ca-cert\") pod \"1c0e3851-c4c8-4bed-971c-97f54574135d\" (UID: \"1c0e3851-c4c8-4bed-971c-97f54574135d\") " Nov 02 06:49:50.584777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.584716 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-data" pod "1c0e3851-c4c8-4bed-971c-97f54574135d" (UID: "1c0e3851-c4c8-4bed-971c-97f54574135d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:49:50.585062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.585035 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0e3851-c4c8-4bed-971c-97f54574135d-istiod-ca-cert" pod "1c0e3851-c4c8-4bed-971c-97f54574135d" (UID: "1c0e3851-c4c8-4bed-971c-97f54574135d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:49:50.585412 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.585389 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:50.585522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.585417 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:50.585522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.585431 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:50.585522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.585445 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:50.585522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.585460 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c0e3851-c4c8-4bed-971c-97f54574135d-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:50.586437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.586411 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-kube-api-access-6zrpz" pod "1c0e3851-c4c8-4bed-971c-97f54574135d" (UID: "1c0e3851-c4c8-4bed-971c-97f54574135d"). InnerVolumeSpecName "kube-api-access-6zrpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:50.586532 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.586472 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-token" pod "1c0e3851-c4c8-4bed-971c-97f54574135d" (UID: "1c0e3851-c4c8-4bed-971c-97f54574135d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:49:50.586532 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.586493 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-envoy" pod "1c0e3851-c4c8-4bed-971c-97f54574135d" (UID: "1c0e3851-c4c8-4bed-971c-97f54574135d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:49:50.586815 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.586797 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-podinfo" pod "1c0e3851-c4c8-4bed-971c-97f54574135d" (UID: "1c0e3851-c4c8-4bed-971c-97f54574135d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:49:50.687613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.687578 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zrpz\" (UniqueName: \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-kube-api-access-6zrpz\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:50.687777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.687620 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:50.687777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.687636 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:50.687777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:50.687648 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c0e3851-c4c8-4bed-971c-97f54574135d-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:49:51.002168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.002129 5025 generic.go:356] "Generic (PLEG): container finished" podID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerID="c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af" exitCode=0 Nov 02 06:49:51.002341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.002178 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" Nov 02 06:49:51.002341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.002216 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" event={"ID":"1c0e3851-c4c8-4bed-971c-97f54574135d","Type":"ContainerDied","Data":"c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af"} Nov 02 06:49:51.002341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.002284 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg" event={"ID":"1c0e3851-c4c8-4bed-971c-97f54574135d","Type":"ContainerDied","Data":"6a3b9fe32a255c78cb63adb2446fb25f308cfd389a3517e58fa9e8724c951c29"} Nov 02 06:49:51.002341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.002309 5025 scope.go:122] "RemoveContainer" containerID="c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af" Nov 02 06:49:51.013512 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.013495 5025 scope.go:122] "RemoveContainer" containerID="976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4" Nov 02 06:49:51.022666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.022651 5025 scope.go:122] "RemoveContainer" containerID="c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af" Nov 02 06:49:51.022909 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:49:51.022893 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af\": container with ID starting with c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af not found: ID does not exist" containerID="c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af" Nov 02 06:49:51.022953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.022915 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af"} err="failed to get container status \"c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af\": rpc error: code = NotFound desc = could not find container \"c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af\": container with ID starting with c0abaccaee164025ecbfc1617a249ed57cb8da05a26e25a3d08108f16ad003af not found: ID does not exist" Nov 02 06:49:51.022953 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.022931 5025 scope.go:122] "RemoveContainer" containerID="976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4" Nov 02 06:49:51.023999 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:49:51.023977 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4\": container with ID starting with 976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4 not found: ID does not exist" containerID="976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4" Nov 02 06:49:51.024100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.024005 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4"} err="failed to get container status \"976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4\": rpc error: code = NotFound desc = could not find container \"976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4\": container with ID starting with 976647723c140a7a1c8146021f3bb9ffc04cdeca31ecc2f10b7af00a997003b4 not found: ID does not exist" Nov 02 06:49:51.161422 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.161388 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg"] Nov 02 06:49:51.208332 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.208304 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-12bfb5-openshift-default-6dc5c5bbbc-mz9lg"] Nov 02 06:49:51.670262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:51.670226 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" path="/var/lib/kubelet/pods/1c0e3851-c4c8-4bed-971c-97f54574135d/volumes" Nov 02 06:49:52.246512 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:52.246475 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:57.346095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:57.341999 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:58.844826 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:58.844796 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5"] Nov 02 06:49:59.325701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:59.325663 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:49:59.511867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:49:59.511829 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:02.117959 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:02.117921 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:02.710163 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:50:02.710126 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:50:03.845317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:03.845279 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" podUID="3b5b8fd5-abb5-439a-90f1-a07a4f9de230" containerName="payload-processing" containerID="cri-o://1147a78049020220f0ebd3be5eed1a9802418a92dbadd8e44ed375fbb2ca25e2" gracePeriod=25 Nov 02 06:50:04.095744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.095669 5025 generic.go:356] "Generic (PLEG): container finished" podID="3b5b8fd5-abb5-439a-90f1-a07a4f9de230" containerID="1147a78049020220f0ebd3be5eed1a9802418a92dbadd8e44ed375fbb2ca25e2" exitCode=0 Nov 02 06:50:04.095896 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.095749 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" event={"ID":"3b5b8fd5-abb5-439a-90f1-a07a4f9de230","Type":"ContainerDied","Data":"1147a78049020220f0ebd3be5eed1a9802418a92dbadd8e44ed375fbb2ca25e2"} Nov 02 06:50:04.095896 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.095848 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" event={"ID":"3b5b8fd5-abb5-439a-90f1-a07a4f9de230","Type":"ContainerDied","Data":"12ec92692c798d542c092bf1aa175b2c5d5e3dadc2020629bbfdb9662dc64329"} Nov 02 06:50:04.095896 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.095871 5025 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12ec92692c798d542c092bf1aa175b2c5d5e3dadc2020629bbfdb9662dc64329" Nov 02 06:50:04.113562 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.113534 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:50:04.249132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.249065 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-plugins-config-volume\") pod \"3b5b8fd5-abb5-439a-90f1-a07a4f9de230\" (UID: \"3b5b8fd5-abb5-439a-90f1-a07a4f9de230\") " Nov 02 06:50:04.249313 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.249214 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-kube-api-access-mb84f\" (UniqueName: \"kubernetes.io/projected/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-kube-api-access-mb84f\") pod \"3b5b8fd5-abb5-439a-90f1-a07a4f9de230\" (UID: \"3b5b8fd5-abb5-439a-90f1-a07a4f9de230\") " Nov 02 06:50:04.249455 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.249432 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-plugins-config-volume" pod "3b5b8fd5-abb5-439a-90f1-a07a4f9de230" (UID: "3b5b8fd5-abb5-439a-90f1-a07a4f9de230"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:04.249995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.249964 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:04.252448 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.252423 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-kube-api-access-mb84f" pod "3b5b8fd5-abb5-439a-90f1-a07a4f9de230" (UID: "3b5b8fd5-abb5-439a-90f1-a07a4f9de230"). InnerVolumeSpecName "kube-api-access-mb84f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:04.341466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.341437 5025 scope.go:122] "RemoveContainer" containerID="1147a78049020220f0ebd3be5eed1a9802418a92dbadd8e44ed375fbb2ca25e2" Nov 02 06:50:04.351511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:04.351367 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb84f\" (UniqueName: \"kubernetes.io/projected/3b5b8fd5-abb5-439a-90f1-a07a4f9de230-kube-api-access-mb84f\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:05.101471 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:05.101441 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5" Nov 02 06:50:05.318108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:05.318063 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5"] Nov 02 06:50:05.322933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:05.322901 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a25b83-7546b5ffdc-pjqs5"] Nov 02 06:50:05.670596 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:05.670561 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b5b8fd5-abb5-439a-90f1-a07a4f9de230" path="/var/lib/kubelet/pods/3b5b8fd5-abb5-439a-90f1-a07a4f9de230/volumes" Nov 02 06:50:06.788030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:06.787994 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:11.391993 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:11.391934 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:15.197862 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.197787 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28"] Nov 02 06:50:15.198312 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.197988 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" podUID="d2fc06f2-607e-44dc-beac-0048d272c8aa" containerName="maas-api" containerID="cri-o://e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334" gracePeriod=30 Nov 02 06:50:15.342894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.342857 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:15.547938 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.547917 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:50:15.649632 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.649593 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fc06f2-607e-44dc-beac-0048d272c8aa-kube-api-access-d79pp\" (UniqueName: \"kubernetes.io/projected/d2fc06f2-607e-44dc-beac-0048d272c8aa-kube-api-access-d79pp\") pod \"d2fc06f2-607e-44dc-beac-0048d272c8aa\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " Nov 02 06:50:15.650444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.650418 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-metrics-certs\") pod \"d2fc06f2-607e-44dc-beac-0048d272c8aa\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " Nov 02 06:50:15.650655 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.650637 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-maas-api-tls\") pod \"d2fc06f2-607e-44dc-beac-0048d272c8aa\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " Nov 02 06:50:15.651047 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.651010 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2fc06f2-607e-44dc-beac-0048d272c8aa-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2fc06f2-607e-44dc-beac-0048d272c8aa-openshift-service-ca\") pod \"d2fc06f2-607e-44dc-beac-0048d272c8aa\" (UID: \"d2fc06f2-607e-44dc-beac-0048d272c8aa\") " Nov 02 06:50:15.657048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.657002 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fc06f2-607e-44dc-beac-0048d272c8aa-kube-api-access-d79pp" pod "d2fc06f2-607e-44dc-beac-0048d272c8aa" (UID: "d2fc06f2-607e-44dc-beac-0048d272c8aa"). InnerVolumeSpecName "kube-api-access-d79pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:15.694492 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.690764 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-maas-api-tls" pod "d2fc06f2-607e-44dc-beac-0048d272c8aa" (UID: "d2fc06f2-607e-44dc-beac-0048d272c8aa"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:50:15.694492 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.691274 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-metrics-certs" pod "d2fc06f2-607e-44dc-beac-0048d272c8aa" (UID: "d2fc06f2-607e-44dc-beac-0048d272c8aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:50:15.694492 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.694372 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fc06f2-607e-44dc-beac-0048d272c8aa-openshift-service-ca" pod "d2fc06f2-607e-44dc-beac-0048d272c8aa" (UID: "d2fc06f2-607e-44dc-beac-0048d272c8aa"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:15.754164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.754128 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d79pp\" (UniqueName: \"kubernetes.io/projected/d2fc06f2-607e-44dc-beac-0048d272c8aa-kube-api-access-d79pp\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:15.754164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.754166 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:15.754369 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.754180 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2fc06f2-607e-44dc-beac-0048d272c8aa-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:15.754369 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.754195 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2fc06f2-607e-44dc-beac-0048d272c8aa-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:15.883578 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.883495 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr"] Nov 02 06:50:15.884210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884194 5025 cpu_manager.go:414] "containerMap: removing container" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" Nov 02 06:50:15.884269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884212 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" Nov 02 06:50:15.884269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884237 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d2fc06f2-607e-44dc-beac-0048d272c8aa" containerName="maas-api" Nov 02 06:50:15.884269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884242 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2fc06f2-607e-44dc-beac-0048d272c8aa" containerName="maas-api" Nov 02 06:50:15.884269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884257 5025 cpu_manager.go:414] "containerMap: removing container" podUID="6aee84db-e7cf-41f6-9e44-e981dacfb8cc" containerName="payload-pre-processing" Nov 02 06:50:15.884269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884263 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aee84db-e7cf-41f6-9e44-e981dacfb8cc" containerName="payload-pre-processing" Nov 02 06:50:15.884431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884278 5025 cpu_manager.go:414] "containerMap: removing container" podUID="3b5b8fd5-abb5-439a-90f1-a07a4f9de230" containerName="payload-processing" Nov 02 06:50:15.884431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884284 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b5b8fd5-abb5-439a-90f1-a07a4f9de230" containerName="payload-processing" Nov 02 06:50:15.884431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884298 5025 cpu_manager.go:414] "containerMap: removing container" podUID="8bd5a095-dd31-4987-bf49-4fd9616a022c" containerName="payload-pre-processing" Nov 02 06:50:15.884431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884303 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd5a095-dd31-4987-bf49-4fd9616a022c" containerName="payload-pre-processing" Nov 02 06:50:15.884431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884315 5025 cpu_manager.go:414] "containerMap: removing container" podUID="99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb" containerName="payload-processing" Nov 02 06:50:15.884431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884320 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb" containerName="payload-processing" Nov 02 06:50:15.884431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884402 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2fc06f2-607e-44dc-beac-0048d272c8aa" containerName="maas-api" Nov 02 06:50:15.884431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884410 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aee84db-e7cf-41f6-9e44-e981dacfb8cc" containerName="payload-pre-processing" Nov 02 06:50:15.884431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884418 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b5b8fd5-abb5-439a-90f1-a07a4f9de230" containerName="payload-processing" Nov 02 06:50:15.884431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884426 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd5a095-dd31-4987-bf49-4fd9616a022c" containerName="payload-pre-processing" Nov 02 06:50:15.884431 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884433 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" Nov 02 06:50:15.884761 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.884439 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f13cb1-b17f-4d7a-ab25-bcc430cd0ceb" containerName="payload-processing" Nov 02 06:50:15.888063 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.888046 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:15.937478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.937451 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3bf02ebf-openshift-default-dockercfg-njbqs\"" Nov 02 06:50:15.957793 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.957760 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:15.957939 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.957901 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:15.957994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.957949 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:15.958130 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.958111 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpg2t\" (UniqueName: \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-kube-api-access-cpg2t\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:15.958217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.958198 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:15.958270 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.958256 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:15.958364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.958348 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:15.958435 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.958420 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:15.958531 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.958515 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:15.968259 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:15.968226 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr"] Nov 02 06:50:16.060234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060204 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.060234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060239 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.060434 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060267 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpg2t\" (UniqueName: \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-kube-api-access-cpg2t\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.060434 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060303 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.060434 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060347 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.060434 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060392 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.060681 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060465 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.060681 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060545 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.060681 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060594 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.061217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060693 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.061217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060698 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.061217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060939 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.061217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.060967 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.061217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.061153 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.062748 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.062724 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.062994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.062978 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.115316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.115274 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpg2t\" (UniqueName: \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-kube-api-access-cpg2t\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.115527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.115504 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.209066 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.207817 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:16.224087 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.221943 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" Nov 02 06:50:16.224087 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.211058 5025 generic.go:356] "Generic (PLEG): container finished" podID="d2fc06f2-607e-44dc-beac-0048d272c8aa" containerID="e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334" exitCode=0 Nov 02 06:50:16.224087 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.223065 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" event={"ID":"d2fc06f2-607e-44dc-beac-0048d272c8aa","Type":"ContainerDied","Data":"e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334"} Nov 02 06:50:16.224087 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.223136 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28" event={"ID":"d2fc06f2-607e-44dc-beac-0048d272c8aa","Type":"ContainerDied","Data":"11172f809ca89149624bc18774a4f507afc5ec31aa64fffa90d2c210182d33c2"} Nov 02 06:50:16.224087 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.223170 5025 scope.go:122] "RemoveContainer" containerID="e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334" Nov 02 06:50:16.311280 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.310877 5025 scope.go:122] "RemoveContainer" containerID="e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334" Nov 02 06:50:16.311374 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:50:16.311300 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334\": container with ID starting with e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334 not found: ID does not exist" containerID="e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334" Nov 02 06:50:16.311374 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.311335 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334"} err="failed to get container status \"e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334\": rpc error: code = NotFound desc = could not find container \"e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334\": container with ID starting with e24a08bfc0b67be767d52e416e5730350b5b2b8f48f3a7b3ac2b76cd8f4cf334 not found: ID does not exist" Nov 02 06:50:16.401506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.401474 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28"] Nov 02 06:50:16.404787 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.404764 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-a25b83-986c97dcc-vvj28"] Nov 02 06:50:16.684451 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.684423 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:16.744846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.744813 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr"] Nov 02 06:50:16.746355 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:50:16.746329 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896cc162_c5d2_45c0_a86a_026df2b1b6f9.slice/crio-88a9c817d557280313edb45c36085489d962a15224d479f428f99166853e682a WatchSource:0}: Error finding container 88a9c817d557280313edb45c36085489d962a15224d479f428f99166853e682a: Status 404 returned error can't find the container with id 88a9c817d557280313edb45c36085489d962a15224d479f428f99166853e682a Nov 02 06:50:16.748967 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.748935 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:16.749046 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.749000 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:16.749046 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.749026 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:16.930583 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:16.930555 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:17.114978 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.114902 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:17.156474 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.156444 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6"] Nov 02 06:50:17.157177 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.157162 5025 cpu_manager.go:414] "containerMap: removing container" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" Nov 02 06:50:17.157245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.157178 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" Nov 02 06:50:17.157299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.157284 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0e3851-c4c8-4bed-971c-97f54574135d" containerName="istio-proxy" Nov 02 06:50:17.160925 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.160902 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.200103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.197253 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36972b30-f157-4a06-aa99-55a5e941643c-istio-podinfo\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.200103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.197305 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-certs\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.200103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.197385 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36972b30-f157-4a06-aa99-55a5e941643c-istiod-ca-cert\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.200103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.197503 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-credential-socket\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.200103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.197575 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-socket\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.200103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.197612 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-envoy\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.200103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.197644 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-istio-token\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.200103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.197691 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-data\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.200103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.197819 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kd7t\" (UniqueName: \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-kube-api-access-7kd7t\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.225098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.216593 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-8bc131-openshift-default-dockercfg-xxdt7\"" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.350876 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-credential-socket\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.350982 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-socket\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.351021 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-envoy\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.351050 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-istio-token\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.351118 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-data\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.351250 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kd7t\" (UniqueName: \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-kube-api-access-7kd7t\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.351297 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36972b30-f157-4a06-aa99-55a5e941643c-istio-podinfo\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.351325 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-certs\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.351406 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36972b30-f157-4a06-aa99-55a5e941643c-istiod-ca-cert\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.352245 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36972b30-f157-4a06-aa99-55a5e941643c-istiod-ca-cert\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.352529 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-credential-socket\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.354095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.352772 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-socket\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.441911 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.402325 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-data\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.441911 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.402597 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-certs\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.460184 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.452755 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:17.484497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.484452 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36972b30-f157-4a06-aa99-55a5e941643c-istio-podinfo\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.496653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.496612 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-envoy\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.497470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.497433 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kd7t\" (UniqueName: \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-kube-api-access-7kd7t\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.500865 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.500835 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-istio-token\") pod \"e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.508135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.508055 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6"] Nov 02 06:50:17.531917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.526860 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" event={"ID":"896cc162-c5d2-45c0-a86a-026df2b1b6f9","Type":"ContainerStarted","Data":"fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7"} Nov 02 06:50:17.531917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.526965 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" event={"ID":"896cc162-c5d2-45c0-a86a-026df2b1b6f9","Type":"ContainerStarted","Data":"88a9c817d557280313edb45c36085489d962a15224d479f428f99166853e682a"} Nov 02 06:50:17.644801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.644725 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" podStartSLOduration=2.64470329 podStartE2EDuration="2.64470329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:50:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:50:17.64256143 +0000 UTC m=+1994.090938185" watchObservedRunningTime="2026-11-02 06:50:17.64470329 +0000 UTC m=+1994.093080050" Nov 02 06:50:17.676561 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.676519 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2fc06f2-607e-44dc-beac-0048d272c8aa" path="/var/lib/kubelet/pods/d2fc06f2-607e-44dc-beac-0048d272c8aa/volumes" Nov 02 06:50:17.723978 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.723898 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:17.775682 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.775649 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:17.928103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:17.926347 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:18.210123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:18.209553 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:18.211379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:18.211346 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 06:50:18.211530 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:18.211403 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" podUID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 06:50:18.311994 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:50:18.311956 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36972b30_f157_4a06_aa99_55a5e941643c.slice/crio-8d7d63c3638483993cedb8419c9ab1a771b7bc547a0ec7906df71123afadbeda WatchSource:0}: Error finding container 8d7d63c3638483993cedb8419c9ab1a771b7bc547a0ec7906df71123afadbeda: Status 404 returned error can't find the container with id 8d7d63c3638483993cedb8419c9ab1a771b7bc547a0ec7906df71123afadbeda Nov 02 06:50:18.315244 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:18.315205 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:18.315331 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:18.315315 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:18.315373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:18.315358 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:18.326832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:18.326802 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6"] Nov 02 06:50:18.582210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:18.582113 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" event={"ID":"36972b30-f157-4a06-aa99-55a5e941643c","Type":"ContainerStarted","Data":"96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c"} Nov 02 06:50:18.582210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:18.582181 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" event={"ID":"36972b30-f157-4a06-aa99-55a5e941643c","Type":"ContainerStarted","Data":"8d7d63c3638483993cedb8419c9ab1a771b7bc547a0ec7906df71123afadbeda"} Nov 02 06:50:18.705317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:18.705255 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" podStartSLOduration=1.705236122 podStartE2EDuration="1.705236122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:50:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:50:18.699390753 +0000 UTC m=+1995.147767538" watchObservedRunningTime="2026-11-02 06:50:18.705236122 +0000 UTC m=+1995.153612876" Nov 02 06:50:18.775966 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:18.775936 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:19.217098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:19.216492 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 06:50:19.217098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:19.216553 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" podUID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 06:50:19.705095 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:19.703476 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:19.782280 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:19.782251 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:19.906646 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:19.906572 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:20.208666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:20.208632 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 06:50:20.208818 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:20.208729 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" podUID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 06:50:20.374100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:20.372620 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:20.602703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:20.602612 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:20.624588 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:20.624547 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:20.625439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:20.625418 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:50:21.209663 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:21.209619 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 06:50:21.210049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:21.209700 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" podUID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 06:50:21.531109 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:21.531015 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:21.725277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:21.725238 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:22.208209 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:22.206602 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:22.209548 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:22.209522 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 06:50:22.209684 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:22.209582 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" podUID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 06:50:23.210048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:23.210010 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" start-of-body= Nov 02 06:50:23.210682 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:23.210648 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" podUID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.130:15021/healthz/ready\": dial tcp 10.133.0.130:15021: connect: connection refused" Nov 02 06:50:24.214274 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:24.214246 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:24.214707 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:24.214693 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:24.215618 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:24.215600 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:50:24.493642 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:24.435017 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:24.691490 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:24.691463 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:24.879629 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:24.879521 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:25.069284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:25.069244 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:25.281702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:25.281598 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:25.518011 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:25.517979 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv"] Nov 02 06:50:25.518219 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:25.518195 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" containerID="cri-o://8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c" gracePeriod=30 Nov 02 06:50:25.533189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:25.533116 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:25.938397 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:25.938363 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:26.185111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:26.184836 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:27.838427 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:27.838395 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:50:27.838781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:27.838454 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:50:29.710160 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:29.709544 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:30.790925 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.790869 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:50:30.947127 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947062 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-certs\") pod \"1f1a9eb4-dde1-488d-96ae-8b699703edab\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " Nov 02 06:50:30.947127 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947127 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-credential-socket\") pod \"1f1a9eb4-dde1-488d-96ae-8b699703edab\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " Nov 02 06:50:30.947387 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947178 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-socket\") pod \"1f1a9eb4-dde1-488d-96ae-8b699703edab\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " Nov 02 06:50:30.947387 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947232 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-podinfo\") pod \"1f1a9eb4-dde1-488d-96ae-8b699703edab\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " Nov 02 06:50:30.947387 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947255 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-data\") pod \"1f1a9eb4-dde1-488d-96ae-8b699703edab\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " Nov 02 06:50:30.947387 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947325 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-envoy\") pod \"1f1a9eb4-dde1-488d-96ae-8b699703edab\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " Nov 02 06:50:30.947387 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947348 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-certs" pod "1f1a9eb4-dde1-488d-96ae-8b699703edab" (UID: "1f1a9eb4-dde1-488d-96ae-8b699703edab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:50:30.947646 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947394 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-kube-api-access-7q5hz\" (UniqueName: \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-kube-api-access-7q5hz\") pod \"1f1a9eb4-dde1-488d-96ae-8b699703edab\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " Nov 02 06:50:30.947646 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947426 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1a9eb4-dde1-488d-96ae-8b699703edab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1a9eb4-dde1-488d-96ae-8b699703edab-istiod-ca-cert\") pod \"1f1a9eb4-dde1-488d-96ae-8b699703edab\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " Nov 02 06:50:30.947646 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947480 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-credential-socket" pod "1f1a9eb4-dde1-488d-96ae-8b699703edab" (UID: "1f1a9eb4-dde1-488d-96ae-8b699703edab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:50:30.947646 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947522 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-socket" pod "1f1a9eb4-dde1-488d-96ae-8b699703edab" (UID: "1f1a9eb4-dde1-488d-96ae-8b699703edab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:50:30.947646 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947551 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-token\" (UniqueName: \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-token\") pod \"1f1a9eb4-dde1-488d-96ae-8b699703edab\" (UID: \"1f1a9eb4-dde1-488d-96ae-8b699703edab\") " Nov 02 06:50:30.947646 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947569 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-data" pod "1f1a9eb4-dde1-488d-96ae-8b699703edab" (UID: "1f1a9eb4-dde1-488d-96ae-8b699703edab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:50:30.947933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.947811 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1a9eb4-dde1-488d-96ae-8b699703edab-istiod-ca-cert" pod "1f1a9eb4-dde1-488d-96ae-8b699703edab" (UID: "1f1a9eb4-dde1-488d-96ae-8b699703edab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:30.948243 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.948222 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:30.948317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.948249 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f1a9eb4-dde1-488d-96ae-8b699703edab-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:30.948317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.948265 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:30.948317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.948279 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:30.948317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.948292 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:30.949387 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.949363 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-podinfo" pod "1f1a9eb4-dde1-488d-96ae-8b699703edab" (UID: "1f1a9eb4-dde1-488d-96ae-8b699703edab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:50:30.949563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.949548 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-kube-api-access-7q5hz" pod "1f1a9eb4-dde1-488d-96ae-8b699703edab" (UID: "1f1a9eb4-dde1-488d-96ae-8b699703edab"). InnerVolumeSpecName "kube-api-access-7q5hz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:30.949871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.949838 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-envoy" pod "1f1a9eb4-dde1-488d-96ae-8b699703edab" (UID: "1f1a9eb4-dde1-488d-96ae-8b699703edab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:50:30.949871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:30.949854 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-token" pod "1f1a9eb4-dde1-488d-96ae-8b699703edab" (UID: "1f1a9eb4-dde1-488d-96ae-8b699703edab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:31.006185 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.006147 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8"] Nov 02 06:50:31.007156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.007132 5025 cpu_manager.go:414] "containerMap: removing container" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" Nov 02 06:50:31.007156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.007153 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" Nov 02 06:50:31.007316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.007174 5025 cpu_manager.go:414] "containerMap: removing container" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" Nov 02 06:50:31.007316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.007182 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" Nov 02 06:50:31.007316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.007305 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" Nov 02 06:50:31.007316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.007316 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerName="istio-proxy" Nov 02 06:50:31.011010 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.010986 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.045347 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.045273 5025 generic.go:356] "Generic (PLEG): container finished" podID="1f1a9eb4-dde1-488d-96ae-8b699703edab" containerID="8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c" exitCode=0 Nov 02 06:50:31.045347 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.045339 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" Nov 02 06:50:31.045544 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.045358 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" event={"ID":"1f1a9eb4-dde1-488d-96ae-8b699703edab","Type":"ContainerDied","Data":"8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c"} Nov 02 06:50:31.045544 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.045421 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv" event={"ID":"1f1a9eb4-dde1-488d-96ae-8b699703edab","Type":"ContainerDied","Data":"e055b81d2c7731389341d3d029562345425570ed9e0a421df91dcd97c27078c2"} Nov 02 06:50:31.045544 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.045444 5025 scope.go:122] "RemoveContainer" containerID="8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c" Nov 02 06:50:31.049457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.049432 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:31.049457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.049456 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q5hz\" (UniqueName: \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-kube-api-access-7q5hz\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:31.049620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.049467 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:31.049620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.049477 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f1a9eb4-dde1-488d-96ae-8b699703edab-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:31.056834 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.056815 5025 scope.go:122] "RemoveContainer" containerID="68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65" Nov 02 06:50:31.066585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.066563 5025 scope.go:122] "RemoveContainer" containerID="8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c" Nov 02 06:50:31.066924 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:50:31.066901 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c\": container with ID starting with 8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c not found: ID does not exist" containerID="8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c" Nov 02 06:50:31.067015 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.066929 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c"} err="failed to get container status \"8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c\": rpc error: code = NotFound desc = could not find container \"8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c\": container with ID starting with 8ef5dee2f8250e3d6d60377fcecb604ca647d7b1bdba78798ec5a244d79f033c not found: ID does not exist" Nov 02 06:50:31.067015 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.066952 5025 scope.go:122] "RemoveContainer" containerID="68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65" Nov 02 06:50:31.067282 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:50:31.067258 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65\": container with ID starting with 68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65 not found: ID does not exist" containerID="68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65" Nov 02 06:50:31.067412 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.067289 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65"} err="failed to get container status \"68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65\": rpc error: code = NotFound desc = could not find container \"68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65\": container with ID starting with 68cd1608de8fbc19a6b64fa0dee98dd1d19d01671624cbcde01efa561b805f65 not found: ID does not exist" Nov 02 06:50:31.115415 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.115383 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8"] Nov 02 06:50:31.150819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.150791 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.150954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.150826 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.150954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.150849 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.150954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.150934 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.151175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.151107 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.151175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.151142 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.151404 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.151382 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.151443 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.151433 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.151554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.151537 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72gsm\" (UniqueName: \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-kube-api-access-72gsm\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.234247 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.234131 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv"] Nov 02 06:50:31.241523 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.241241 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-a25b83-openshift-default-587fb458b9-cdhkv"] Nov 02 06:50:31.252531 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.252494 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.252686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.252539 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.252686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.252578 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.252786 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.252716 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.252786 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.252778 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.252909 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.252800 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.252968 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.252943 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.253028 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.252978 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.253028 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.252998 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-credential-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.253147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.253033 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-socket\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.253147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.253109 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-data\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.253147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.253138 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72gsm\" (UniqueName: \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-kube-api-access-72gsm\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.253474 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.253453 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-certs\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.253635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.253615 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istiod-ca-cert\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.255160 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.255143 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-envoy\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.255375 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.255357 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-podinfo\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.306940 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.306885 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-token\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.307049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.307031 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72gsm\" (UniqueName: \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-kube-api-access-72gsm\") pod \"e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.324769 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.324744 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:31.671301 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.671261 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1a9eb4-dde1-488d-96ae-8b699703edab" path="/var/lib/kubelet/pods/1f1a9eb4-dde1-488d-96ae-8b699703edab/volumes" Nov 02 06:50:31.744711 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:50:31.744680 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2637f05a_9e43_47c5_9e2c_8bffb1e58e4b.slice/crio-4a61e758f8c9bc56a4f0e7f7167d08fb8742f260e0fc0f4fff4c855c3cb27b79 WatchSource:0}: Error finding container 4a61e758f8c9bc56a4f0e7f7167d08fb8742f260e0fc0f4fff4c855c3cb27b79: Status 404 returned error can't find the container with id 4a61e758f8c9bc56a4f0e7f7167d08fb8742f260e0fc0f4fff4c855c3cb27b79 Nov 02 06:50:31.744847 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.744809 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8"] Nov 02 06:50:31.747204 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.747173 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:31.747288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.747237 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:31.747288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.747267 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:50:31.973158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:31.972644 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:32.055145 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:32.055109 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" event={"ID":"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b","Type":"ContainerStarted","Data":"628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d"} Nov 02 06:50:32.055293 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:32.055164 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" event={"ID":"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b","Type":"ContainerStarted","Data":"4a61e758f8c9bc56a4f0e7f7167d08fb8742f260e0fc0f4fff4c855c3cb27b79"} Nov 02 06:50:32.173478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:32.173443 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:32.341896 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:32.335423 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:32.356098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:32.347191 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 06:50:32.356098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:32.347254 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" podUID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 06:50:32.386096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:32.385992 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:32.722051 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:50:32.722015 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:50:33.325425 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:33.325386 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 06:50:33.325925 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:33.325457 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" podUID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 06:50:34.326182 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:34.326145 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 06:50:34.326656 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:34.326205 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" podUID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 06:50:34.685994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:34.685762 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:35.325578 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:35.325540 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 06:50:35.325759 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:35.325598 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" podUID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 06:50:36.325685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:36.325650 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 06:50:36.326072 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:36.325702 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" podUID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 06:50:37.325385 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:37.325351 5025 patch_prober.go:29] interesting pod/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Nov 02 06:50:37.325570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:37.325404 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" podUID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Nov 02 06:50:38.208283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:38.208225 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:38.332806 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:38.332774 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:38.333317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:38.333287 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:38.334148 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:38.334126 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:50:38.555818 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:38.555720 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" podStartSLOduration=8.555707568999999 podStartE2EDuration="8.555707569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:50:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:50:32.206331429 +0000 UTC m=+2008.654708184" watchObservedRunningTime="2026-11-02 06:50:38.555707569 +0000 UTC m=+2015.004084323" Nov 02 06:50:38.565117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:38.565054 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:39.756435 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:39.756383 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:44.400976 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:44.400914 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:44.433270 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:44.433233 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl"] Nov 02 06:50:44.790821 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:44.790746 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:44.973203 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:44.972771 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:49.163960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:49.163920 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:49.434478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:49.434390 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" podUID="dfc762bd-184c-428f-b6e4-7fd135b57154" containerName="payload-processing" containerID="cri-o://7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1" gracePeriod=25 Nov 02 06:50:49.712295 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:49.712270 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:50:49.751227 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:49.751202 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfc762bd-184c-428f-b6e4-7fd135b57154-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc762bd-184c-428f-b6e4-7fd135b57154-plugins-config-volume\") pod \"dfc762bd-184c-428f-b6e4-7fd135b57154\" (UID: \"dfc762bd-184c-428f-b6e4-7fd135b57154\") " Nov 02 06:50:49.751376 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:49.751344 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfc762bd-184c-428f-b6e4-7fd135b57154-kube-api-access-fvptn\" (UniqueName: \"kubernetes.io/projected/dfc762bd-184c-428f-b6e4-7fd135b57154-kube-api-access-fvptn\") pod \"dfc762bd-184c-428f-b6e4-7fd135b57154\" (UID: \"dfc762bd-184c-428f-b6e4-7fd135b57154\") " Nov 02 06:50:49.751557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:49.751534 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc762bd-184c-428f-b6e4-7fd135b57154-plugins-config-volume" pod "dfc762bd-184c-428f-b6e4-7fd135b57154" (UID: "dfc762bd-184c-428f-b6e4-7fd135b57154"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:50:49.751868 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:49.751852 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dfc762bd-184c-428f-b6e4-7fd135b57154-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:49.753477 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:49.753453 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc762bd-184c-428f-b6e4-7fd135b57154-kube-api-access-fvptn" pod "dfc762bd-184c-428f-b6e4-7fd135b57154" (UID: "dfc762bd-184c-428f-b6e4-7fd135b57154"). InnerVolumeSpecName "kube-api-access-fvptn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:50:49.854042 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:49.854006 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvptn\" (UniqueName: \"kubernetes.io/projected/dfc762bd-184c-428f-b6e4-7fd135b57154-kube-api-access-fvptn\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:50:50.221811 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:50.221777 5025 generic.go:356] "Generic (PLEG): container finished" podID="dfc762bd-184c-428f-b6e4-7fd135b57154" containerID="7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1" exitCode=0 Nov 02 06:50:50.222267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:50.221827 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" Nov 02 06:50:50.222267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:50.221826 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" event={"ID":"dfc762bd-184c-428f-b6e4-7fd135b57154","Type":"ContainerDied","Data":"7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1"} Nov 02 06:50:50.222267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:50.221956 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl" event={"ID":"dfc762bd-184c-428f-b6e4-7fd135b57154","Type":"ContainerDied","Data":"a1af6c0c96215bf31d5c45ba7ddf2175ce250714a961727730cb1c20dfe1579a"} Nov 02 06:50:50.222267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:50.221983 5025 scope.go:122] "RemoveContainer" containerID="7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1" Nov 02 06:50:50.232296 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:50.232270 5025 scope.go:122] "RemoveContainer" containerID="7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1" Nov 02 06:50:50.232582 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:50:50.232563 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1\": container with ID starting with 7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1 not found: ID does not exist" containerID="7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1" Nov 02 06:50:50.232642 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:50.232592 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1"} err="failed to get container status \"7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1\": rpc error: code = NotFound desc = could not find container \"7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1\": container with ID starting with 7ee33f444fb18b3c5ef8dd9381cb33cf171e1822e5d397d1f966ae51679fbcd1 not found: ID does not exist" Nov 02 06:50:50.381206 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:50.381173 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl"] Nov 02 06:50:50.386099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:50.386057 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-6d13c3-7b6c6d8499-pzrfl"] Nov 02 06:50:51.670626 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:51.670591 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfc762bd-184c-428f-b6e4-7fd135b57154" path="/var/lib/kubelet/pods/dfc762bd-184c-428f-b6e4-7fd135b57154/volumes" Nov 02 06:50:54.119735 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:54.119654 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:50:58.694144 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:50:58.693823 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:02.300376 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:02.300340 5025 generic.go:356] "Generic (PLEG): container finished" podID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerID="896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe" exitCode=0 Nov 02 06:51:02.300934 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:02.300415 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" event={"ID":"90a63809-f2cc-4157-80e4-d5cba8ddb07d","Type":"ContainerDied","Data":"896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe"} Nov 02 06:51:02.301015 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:02.300953 5025 scope.go:122] "RemoveContainer" containerID="896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe" Nov 02 06:51:02.302564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:02.302529 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:51:02.302696 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:02.302614 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:51:02.302696 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:02.302660 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:51:02.713561 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:51:02.713529 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:51:03.315552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:03.315521 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" event={"ID":"90a63809-f2cc-4157-80e4-d5cba8ddb07d","Type":"ContainerStarted","Data":"1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed"} Nov 02 06:51:03.614578 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:03.614469 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:04.250565 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:04.250531 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:51:04.251758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:04.251731 5025 patch_prober.go:29] interesting pod/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:51:04.251897 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:04.251782 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 06:51:04.273958 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:04.273932 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:51:05.250808 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:05.250771 5025 patch_prober.go:29] interesting pod/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:51:05.251298 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:05.250824 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 06:51:06.250474 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:06.250436 5025 patch_prober.go:29] interesting pod/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:51:06.250641 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:06.250499 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 06:51:07.063428 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:07.063391 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2"] Nov 02 06:51:07.063949 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:07.063624 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" podUID="3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" containerName="istio-proxy" containerID="cri-o://bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987" gracePeriod=30 Nov 02 06:51:07.251098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:07.251039 5025 patch_prober.go:29] interesting pod/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:51:07.251098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:07.251099 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 06:51:08.143631 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:08.143588 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf"] Nov 02 06:51:08.144062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:08.143826 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" containerID="cri-o://d01ba258a7ecc282fba60584316d3119e67870563442c438ff4e98624195ecff" gracePeriod=30 Nov 02 06:51:08.252357 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:08.251847 5025 patch_prober.go:29] interesting pod/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" start-of-body= Nov 02 06:51:08.252357 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:08.251922 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.129:15021/healthz/ready\": dial tcp 10.133.0.129:15021: connect: connection refused" Nov 02 06:51:08.540859 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:08.540716 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:09.254405 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:09.254376 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:51:09.255597 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:09.255575 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 06:51:12.326491 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.326427 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:51:12.382963 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.382932 5025 generic.go:356] "Generic (PLEG): container finished" podID="3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" containerID="bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987" exitCode=0 Nov 02 06:51:12.383165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.382979 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" Nov 02 06:51:12.383165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.383012 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" event={"ID":"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9","Type":"ContainerDied","Data":"bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987"} Nov 02 06:51:12.383165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.383066 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2" event={"ID":"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9","Type":"ContainerDied","Data":"3e20df19d4c5f018c1cb3aa3945e6adee218ef2b8ae97d5715c80a33caea1f8d"} Nov 02 06:51:12.383165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.383122 5025 scope.go:122] "RemoveContainer" containerID="bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987" Nov 02 06:51:12.392567 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.392543 5025 scope.go:122] "RemoveContainer" containerID="bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987" Nov 02 06:51:12.392815 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:51:12.392800 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987\": container with ID starting with bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987 not found: ID does not exist" containerID="bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987" Nov 02 06:51:12.392858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.392826 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987"} err="failed to get container status \"bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987\": rpc error: code = NotFound desc = could not find container \"bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987\": container with ID starting with bd0dcecbca244e1766eec133294a8b55bed3096ddaa1b9b85a5afbba5b012987 not found: ID does not exist" Nov 02 06:51:12.394352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.394331 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istiod-ca-cert\") pod \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " Nov 02 06:51:12.394464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.394452 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-token\" (UniqueName: \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-token\") pod \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " Nov 02 06:51:12.394507 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.394479 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-socket\") pod \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " Nov 02 06:51:12.394554 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.394543 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-envoy\") pod \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " Nov 02 06:51:12.394640 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.394628 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-credential-socket\") pod \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " Nov 02 06:51:12.394718 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.394671 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-podinfo\") pod \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " Nov 02 06:51:12.394718 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.394692 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istiod-ca-cert" pod "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" (UID: "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:12.394718 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.394701 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-certs\") pod \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " Nov 02 06:51:12.394902 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.394840 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-socket" pod "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" (UID: "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:12.394902 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.394760 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-data\") pod \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " Nov 02 06:51:12.395018 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.394900 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-certs" pod "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" (UID: "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:12.395108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.395025 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-credential-socket" pod "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" (UID: "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:12.395171 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.395151 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-data" pod "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" (UID: "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:12.395508 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.395488 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.395508 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.395508 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.395669 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.395516 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.395669 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.395524 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.395669 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.395532 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.396704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.396682 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-envoy" pod "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" (UID: "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:12.396809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.396794 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-token" pod "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" (UID: "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:12.396864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.396847 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-podinfo" pod "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" (UID: "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:51:12.496841 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.496812 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-kube-api-access-zz5v4\" (UniqueName: \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-kube-api-access-zz5v4\") pod \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\" (UID: \"3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9\") " Nov 02 06:51:12.497563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.497533 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.497563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.497555 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.497563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.497565 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.499295 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.499271 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-kube-api-access-zz5v4" pod "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" (UID: "3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9"). InnerVolumeSpecName "kube-api-access-zz5v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:12.598822 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.598743 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zz5v4\" (UniqueName: \"kubernetes.io/projected/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9-kube-api-access-zz5v4\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:12.800445 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.800416 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2"] Nov 02 06:51:12.804481 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:12.804455 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-fnrs2"] Nov 02 06:51:13.408024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.407946 5025 generic.go:356] "Generic (PLEG): container finished" podID="8b39f394-283b-4454-8ec6-485f30315d32" containerID="d01ba258a7ecc282fba60584316d3119e67870563442c438ff4e98624195ecff" exitCode=0 Nov 02 06:51:13.408024 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.408000 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" event={"ID":"8b39f394-283b-4454-8ec6-485f30315d32","Type":"ContainerDied","Data":"d01ba258a7ecc282fba60584316d3119e67870563442c438ff4e98624195ecff"} Nov 02 06:51:13.437438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.437416 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:51:13.510749 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.510719 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-envoy\") pod \"8b39f394-283b-4454-8ec6-485f30315d32\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " Nov 02 06:51:13.510877 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.510773 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-socket\") pod \"8b39f394-283b-4454-8ec6-485f30315d32\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " Nov 02 06:51:13.510877 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.510820 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-istio-token\" (UniqueName: \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-istio-token\") pod \"8b39f394-283b-4454-8ec6-485f30315d32\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " Nov 02 06:51:13.510877 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.510846 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-credential-socket\") pod \"8b39f394-283b-4454-8ec6-485f30315d32\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " Nov 02 06:51:13.511028 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.510920 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-certs\") pod \"8b39f394-283b-4454-8ec6-485f30315d32\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " Nov 02 06:51:13.511028 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.510968 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-data\") pod \"8b39f394-283b-4454-8ec6-485f30315d32\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " Nov 02 06:51:13.511147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511018 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-kube-api-access-lkvkg\" (UniqueName: \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-kube-api-access-lkvkg\") pod \"8b39f394-283b-4454-8ec6-485f30315d32\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " Nov 02 06:51:13.511147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511094 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b39f394-283b-4454-8ec6-485f30315d32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b39f394-283b-4454-8ec6-485f30315d32-istiod-ca-cert\") pod \"8b39f394-283b-4454-8ec6-485f30315d32\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " Nov 02 06:51:13.511240 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511214 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-credential-socket" pod "8b39f394-283b-4454-8ec6-485f30315d32" (UID: "8b39f394-283b-4454-8ec6-485f30315d32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:13.511286 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511235 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-certs" pod "8b39f394-283b-4454-8ec6-485f30315d32" (UID: "8b39f394-283b-4454-8ec6-485f30315d32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:13.511286 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511256 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b39f394-283b-4454-8ec6-485f30315d32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b39f394-283b-4454-8ec6-485f30315d32-istio-podinfo\") pod \"8b39f394-283b-4454-8ec6-485f30315d32\" (UID: \"8b39f394-283b-4454-8ec6-485f30315d32\") " Nov 02 06:51:13.511397 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511340 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-data" pod "8b39f394-283b-4454-8ec6-485f30315d32" (UID: "8b39f394-283b-4454-8ec6-485f30315d32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:13.511571 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511546 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b39f394-283b-4454-8ec6-485f30315d32-istiod-ca-cert" pod "8b39f394-283b-4454-8ec6-485f30315d32" (UID: "8b39f394-283b-4454-8ec6-485f30315d32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:13.511710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511664 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-socket" pod "8b39f394-283b-4454-8ec6-485f30315d32" (UID: "8b39f394-283b-4454-8ec6-485f30315d32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:13.511974 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511951 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.512117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511978 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.512117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511991 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.512117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.511999 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b39f394-283b-4454-8ec6-485f30315d32-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.512117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.512013 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.513284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.513261 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-envoy" pod "8b39f394-283b-4454-8ec6-485f30315d32" (UID: "8b39f394-283b-4454-8ec6-485f30315d32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:13.513512 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.513496 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-istio-token" pod "8b39f394-283b-4454-8ec6-485f30315d32" (UID: "8b39f394-283b-4454-8ec6-485f30315d32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:13.513570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.513540 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-kube-api-access-lkvkg" pod "8b39f394-283b-4454-8ec6-485f30315d32" (UID: "8b39f394-283b-4454-8ec6-485f30315d32"). InnerVolumeSpecName "kube-api-access-lkvkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:13.513570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.513556 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b39f394-283b-4454-8ec6-485f30315d32-istio-podinfo" pod "8b39f394-283b-4454-8ec6-485f30315d32" (UID: "8b39f394-283b-4454-8ec6-485f30315d32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:51:13.627396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.615732 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b39f394-283b-4454-8ec6-485f30315d32-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.627396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.615767 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.627396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.615797 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkvkg\" (UniqueName: \"kubernetes.io/projected/8b39f394-283b-4454-8ec6-485f30315d32-kube-api-access-lkvkg\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.627396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.615812 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b39f394-283b-4454-8ec6-485f30315d32-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:13.627396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.620785 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:13.689374 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.689338 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" path="/var/lib/kubelet/pods/3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9/volumes" Nov 02 06:51:13.703546 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.703518 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj"] Nov 02 06:51:13.806041 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:13.806007 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn"] Nov 02 06:51:14.130149 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:14.126942 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:14.416202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:14.416176 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" Nov 02 06:51:14.416772 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:14.416209 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf" event={"ID":"8b39f394-283b-4454-8ec6-485f30315d32","Type":"ContainerDied","Data":"43daa11731926422cb1ffebe0167cc378a5e39e6d997c05ac14df9f9820abc11"} Nov 02 06:51:14.416772 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:14.416273 5025 scope.go:122] "RemoveContainer" containerID="d01ba258a7ecc282fba60584316d3119e67870563442c438ff4e98624195ecff" Nov 02 06:51:14.464659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:14.464631 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:14.578263 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:14.578234 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf"] Nov 02 06:51:14.582277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:14.582247 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-cchpf"] Nov 02 06:51:15.669961 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:15.669920 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b39f394-283b-4454-8ec6-485f30315d32" path="/var/lib/kubelet/pods/8b39f394-283b-4454-8ec6-485f30315d32/volumes" Nov 02 06:51:18.556776 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:18.556683 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:18.704401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:18.704350 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" podUID="60030f97-89f9-4b64-980f-787c1e4c832a" containerName="payload-processing" containerID="cri-o://f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c" gracePeriod=25 Nov 02 06:51:18.806665 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:18.806633 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" podUID="aa8bbf32-36d7-497f-8deb-c8afebfe67d4" containerName="payload-pre-processing" containerID="cri-o://718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8" gracePeriod=25 Nov 02 06:51:18.975769 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:18.975741 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:51:19.064201 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.064113 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60030f97-89f9-4b64-980f-787c1e4c832a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60030f97-89f9-4b64-980f-787c1e4c832a-plugins-config-volume\") pod \"60030f97-89f9-4b64-980f-787c1e4c832a\" (UID: \"60030f97-89f9-4b64-980f-787c1e4c832a\") " Nov 02 06:51:19.064352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.064333 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60030f97-89f9-4b64-980f-787c1e4c832a-kube-api-access-m8dc6\" (UniqueName: \"kubernetes.io/projected/60030f97-89f9-4b64-980f-787c1e4c832a-kube-api-access-m8dc6\") pod \"60030f97-89f9-4b64-980f-787c1e4c832a\" (UID: \"60030f97-89f9-4b64-980f-787c1e4c832a\") " Nov 02 06:51:19.064545 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.064521 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60030f97-89f9-4b64-980f-787c1e4c832a-plugins-config-volume" pod "60030f97-89f9-4b64-980f-787c1e4c832a" (UID: "60030f97-89f9-4b64-980f-787c1e4c832a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:19.064971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.064950 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60030f97-89f9-4b64-980f-787c1e4c832a-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:19.066648 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.066625 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60030f97-89f9-4b64-980f-787c1e4c832a-kube-api-access-m8dc6" pod "60030f97-89f9-4b64-980f-787c1e4c832a" (UID: "60030f97-89f9-4b64-980f-787c1e4c832a"). InnerVolumeSpecName "kube-api-access-m8dc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:19.098172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.098144 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:51:19.166352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.166319 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-plugins-config-volume\") pod \"aa8bbf32-36d7-497f-8deb-c8afebfe67d4\" (UID: \"aa8bbf32-36d7-497f-8deb-c8afebfe67d4\") " Nov 02 06:51:19.166643 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.166626 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-kube-api-access-m7gph\" (UniqueName: \"kubernetes.io/projected/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-kube-api-access-m7gph\") pod \"aa8bbf32-36d7-497f-8deb-c8afebfe67d4\" (UID: \"aa8bbf32-36d7-497f-8deb-c8afebfe67d4\") " Nov 02 06:51:19.166715 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.166648 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-plugins-config-volume" pod "aa8bbf32-36d7-497f-8deb-c8afebfe67d4" (UID: "aa8bbf32-36d7-497f-8deb-c8afebfe67d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:19.167128 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.167109 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8dc6\" (UniqueName: \"kubernetes.io/projected/60030f97-89f9-4b64-980f-787c1e4c832a-kube-api-access-m8dc6\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:19.167221 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.167131 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:19.168639 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.168621 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-kube-api-access-m7gph" pod "aa8bbf32-36d7-497f-8deb-c8afebfe67d4" (UID: "aa8bbf32-36d7-497f-8deb-c8afebfe67d4"). InnerVolumeSpecName "kube-api-access-m7gph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:19.268545 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.268514 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7gph\" (UniqueName: \"kubernetes.io/projected/aa8bbf32-36d7-497f-8deb-c8afebfe67d4-kube-api-access-m7gph\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:19.500602 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.500569 5025 generic.go:356] "Generic (PLEG): container finished" podID="aa8bbf32-36d7-497f-8deb-c8afebfe67d4" containerID="718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8" exitCode=0 Nov 02 06:51:19.500755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.500611 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" Nov 02 06:51:19.500755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.500655 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" event={"ID":"aa8bbf32-36d7-497f-8deb-c8afebfe67d4","Type":"ContainerDied","Data":"718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8"} Nov 02 06:51:19.500755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.500708 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn" event={"ID":"aa8bbf32-36d7-497f-8deb-c8afebfe67d4","Type":"ContainerDied","Data":"34dd4b648c012416a2d4c9d0a9c0238a3e3141c4abd22c69b1320b92b1f2bfa7"} Nov 02 06:51:19.500755 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.500735 5025 scope.go:122] "RemoveContainer" containerID="718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8" Nov 02 06:51:19.502051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.502028 5025 generic.go:356] "Generic (PLEG): container finished" podID="60030f97-89f9-4b64-980f-787c1e4c832a" containerID="f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c" exitCode=0 Nov 02 06:51:19.502165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.502105 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" event={"ID":"60030f97-89f9-4b64-980f-787c1e4c832a","Type":"ContainerDied","Data":"f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c"} Nov 02 06:51:19.502165 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.502117 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" Nov 02 06:51:19.502278 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.502164 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj" event={"ID":"60030f97-89f9-4b64-980f-787c1e4c832a","Type":"ContainerDied","Data":"961f54c3b0651d05e0386408768c482be0b6d0985cdb82da28e51558ca11a48a"} Nov 02 06:51:19.510975 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.510956 5025 scope.go:122] "RemoveContainer" containerID="718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8" Nov 02 06:51:19.511305 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:51:19.511281 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8\": container with ID starting with 718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8 not found: ID does not exist" containerID="718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8" Nov 02 06:51:19.511400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.511313 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8"} err="failed to get container status \"718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8\": rpc error: code = NotFound desc = could not find container \"718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8\": container with ID starting with 718c2dc8513262e5044f9e59f45f2b8f527778dfeb0b5b5904b31170211bade8 not found: ID does not exist" Nov 02 06:51:19.511400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.511331 5025 scope.go:122] "RemoveContainer" containerID="f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c" Nov 02 06:51:19.520620 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.520601 5025 scope.go:122] "RemoveContainer" containerID="f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c" Nov 02 06:51:19.520856 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:51:19.520840 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c\": container with ID starting with f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c not found: ID does not exist" containerID="f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c" Nov 02 06:51:19.520901 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.520862 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c"} err="failed to get container status \"f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c\": rpc error: code = NotFound desc = could not find container \"f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c\": container with ID starting with f88567a21bc484fde04f8c8fc95bf4c6e637e94a059b6d2b57e6df8629a5582c not found: ID does not exist" Nov 02 06:51:19.670039 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.670010 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn"] Nov 02 06:51:19.709215 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.709190 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3bf02ebf-5c67876455-h67dn"] Nov 02 06:51:19.813819 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.813737 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj"] Nov 02 06:51:19.816651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:19.816627 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3bf02ebf-99d454db7-4ftcj"] Nov 02 06:51:21.669288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:21.669255 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60030f97-89f9-4b64-980f-787c1e4c832a" path="/var/lib/kubelet/pods/60030f97-89f9-4b64-980f-787c1e4c832a/volumes" Nov 02 06:51:21.669772 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:21.669753 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8bbf32-36d7-497f-8deb-c8afebfe67d4" path="/var/lib/kubelet/pods/aa8bbf32-36d7-497f-8deb-c8afebfe67d4/volumes" Nov 02 06:51:23.406223 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:23.405805 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:25.280581 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:25.280544 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:27.496317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:27.496281 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:28.967146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:28.967097 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:29.204490 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:29.204440 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:29.403147 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:29.401571 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:29.869188 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:29.869155 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:32.707500 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:51:32.707465 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:51:34.196145 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:34.195961 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:36.342634 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:36.342598 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:36.548709 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:36.548675 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:36.594527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:36.594448 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9"] Nov 02 06:51:36.594686 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:36.594663 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" containerID="cri-o://8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a" gracePeriod=30 Nov 02 06:51:36.751157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:36.748628 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:38.592172 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:38.592140 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:38.785342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:38.785293 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:38.971855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:38.971808 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:41.873818 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.873764 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:51:41.960850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.960823 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-certs\") pod \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " Nov 02 06:51:41.961023 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.960861 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-kube-api-access-rfcmn\" (UniqueName: \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-kube-api-access-rfcmn\") pod \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " Nov 02 06:51:41.961023 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.960882 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-token\" (UniqueName: \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-token\") pod \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " Nov 02 06:51:41.961023 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.960916 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-credential-socket\") pod \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " Nov 02 06:51:41.961212 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.961034 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-envoy\") pod \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " Nov 02 06:51:41.961212 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.961052 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-podinfo\") pod \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " Nov 02 06:51:41.961212 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.961089 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-socket\") pod \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " Nov 02 06:51:41.961212 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.961150 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-certs" pod "8e296f80-34b3-44ad-b978-bcd9b99f19c1" (UID: "8e296f80-34b3-44ad-b978-bcd9b99f19c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:41.961438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.961254 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istiod-ca-cert\") pod \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " Nov 02 06:51:41.961438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.961294 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-data\") pod \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\" (UID: \"8e296f80-34b3-44ad-b978-bcd9b99f19c1\") " Nov 02 06:51:41.961438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.961317 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-credential-socket" pod "8e296f80-34b3-44ad-b978-bcd9b99f19c1" (UID: "8e296f80-34b3-44ad-b978-bcd9b99f19c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:41.961438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.961391 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-socket" pod "8e296f80-34b3-44ad-b978-bcd9b99f19c1" (UID: "8e296f80-34b3-44ad-b978-bcd9b99f19c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:41.961661 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.961620 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-data" pod "8e296f80-34b3-44ad-b978-bcd9b99f19c1" (UID: "8e296f80-34b3-44ad-b978-bcd9b99f19c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:41.961720 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.961682 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istiod-ca-cert" pod "8e296f80-34b3-44ad-b978-bcd9b99f19c1" (UID: "8e296f80-34b3-44ad-b978-bcd9b99f19c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:41.961994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.961972 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:41.962116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.962002 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:41.962116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.962018 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:41.962116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.962032 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:41.962116 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.962046 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:41.963264 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.963238 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-podinfo" pod "8e296f80-34b3-44ad-b978-bcd9b99f19c1" (UID: "8e296f80-34b3-44ad-b978-bcd9b99f19c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:51:41.963355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.963268 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-token" pod "8e296f80-34b3-44ad-b978-bcd9b99f19c1" (UID: "8e296f80-34b3-44ad-b978-bcd9b99f19c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:41.963508 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.963489 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-envoy" pod "8e296f80-34b3-44ad-b978-bcd9b99f19c1" (UID: "8e296f80-34b3-44ad-b978-bcd9b99f19c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:51:41.963629 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:41.963519 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-kube-api-access-rfcmn" pod "8e296f80-34b3-44ad-b978-bcd9b99f19c1" (UID: "8e296f80-34b3-44ad-b978-bcd9b99f19c1"). InnerVolumeSpecName "kube-api-access-rfcmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:42.063919 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.063880 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfcmn\" (UniqueName: \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-kube-api-access-rfcmn\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:42.063919 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.063915 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:42.064203 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.063931 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:42.064203 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.063944 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e296f80-34b3-44ad-b978-bcd9b99f19c1-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:42.684199 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.684167 5025 generic.go:356] "Generic (PLEG): container finished" podID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerID="8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a" exitCode=0 Nov 02 06:51:42.684373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.684205 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" Nov 02 06:51:42.684373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.684247 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" event={"ID":"8e296f80-34b3-44ad-b978-bcd9b99f19c1","Type":"ContainerDied","Data":"8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a"} Nov 02 06:51:42.684373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.684296 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9" event={"ID":"8e296f80-34b3-44ad-b978-bcd9b99f19c1","Type":"ContainerDied","Data":"2c3bdad613707541a23f50cf6c1a4470d24bcaad2ce01277622f3130bdd35b98"} Nov 02 06:51:42.684373 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.684319 5025 scope.go:122] "RemoveContainer" containerID="8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a" Nov 02 06:51:42.694804 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.694788 5025 scope.go:122] "RemoveContainer" containerID="efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244" Nov 02 06:51:42.703569 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.703553 5025 scope.go:122] "RemoveContainer" containerID="8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a" Nov 02 06:51:42.703815 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:51:42.703793 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a\": container with ID starting with 8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a not found: ID does not exist" containerID="8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a" Nov 02 06:51:42.703898 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.703825 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a"} err="failed to get container status \"8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a\": rpc error: code = NotFound desc = could not find container \"8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a\": container with ID starting with 8b2698b09b8c998e77232633593a6b7f36b9b5c52885be4b1b7a2a557f28479a not found: ID does not exist" Nov 02 06:51:42.703898 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.703846 5025 scope.go:122] "RemoveContainer" containerID="efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244" Nov 02 06:51:42.704091 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:51:42.704061 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244\": container with ID starting with efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244 not found: ID does not exist" containerID="efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244" Nov 02 06:51:42.704159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.704099 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244"} err="failed to get container status \"efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244\": rpc error: code = NotFound desc = could not find container \"efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244\": container with ID starting with efe92cc7347c2a864cd74b426f09d7af35a701b212b62ab4dc8ae0aef74a9244 not found: ID does not exist" Nov 02 06:51:42.843070 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.843043 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9"] Nov 02 06:51:42.890476 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:42.890443 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8ecb23-openshift-default-76484f54f9-jzwv9"] Nov 02 06:51:43.670675 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:43.670638 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" path="/var/lib/kubelet/pods/8e296f80-34b3-44ad-b978-bcd9b99f19c1/volumes" Nov 02 06:51:43.944328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:43.942648 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:44.126816 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:44.126779 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:44.309521 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:44.309442 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:44.880423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:44.880144 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:45.296113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.296054 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:45.730514 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.730485 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr"] Nov 02 06:51:45.731188 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731171 5025 cpu_manager.go:414] "containerMap: removing container" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" Nov 02 06:51:45.731188 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731189 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" Nov 02 06:51:45.731283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731206 5025 cpu_manager.go:414] "containerMap: removing container" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" Nov 02 06:51:45.731283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731212 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" Nov 02 06:51:45.731283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731225 5025 cpu_manager.go:414] "containerMap: removing container" podUID="dfc762bd-184c-428f-b6e4-7fd135b57154" containerName="payload-processing" Nov 02 06:51:45.731283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731230 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfc762bd-184c-428f-b6e4-7fd135b57154" containerName="payload-processing" Nov 02 06:51:45.731283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731257 5025 cpu_manager.go:414] "containerMap: removing container" podUID="3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" containerName="istio-proxy" Nov 02 06:51:45.731283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731261 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" containerName="istio-proxy" Nov 02 06:51:45.731283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731270 5025 cpu_manager.go:414] "containerMap: removing container" podUID="60030f97-89f9-4b64-980f-787c1e4c832a" containerName="payload-processing" Nov 02 06:51:45.731283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731274 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60030f97-89f9-4b64-980f-787c1e4c832a" containerName="payload-processing" Nov 02 06:51:45.731283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731281 5025 cpu_manager.go:414] "containerMap: removing container" podUID="aa8bbf32-36d7-497f-8deb-c8afebfe67d4" containerName="payload-pre-processing" Nov 02 06:51:45.731283 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731285 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8bbf32-36d7-497f-8deb-c8afebfe67d4" containerName="payload-pre-processing" Nov 02 06:51:45.731585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731369 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" Nov 02 06:51:45.731585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731378 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfc762bd-184c-428f-b6e4-7fd135b57154" containerName="payload-processing" Nov 02 06:51:45.731585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731389 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" Nov 02 06:51:45.731585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731394 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ececdd4-fb2d-41d4-b0e1-1d924a9c21e9" containerName="istio-proxy" Nov 02 06:51:45.731585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731400 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="60030f97-89f9-4b64-980f-787c1e4c832a" containerName="payload-processing" Nov 02 06:51:45.731585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731406 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8bbf32-36d7-497f-8deb-c8afebfe67d4" containerName="payload-pre-processing" Nov 02 06:51:45.731585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.731413 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b39f394-283b-4454-8ec6-485f30315d32" containerName="istio-proxy" Nov 02 06:51:45.736197 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.736183 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.787034 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.787000 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5a85b0-openshift-default-dockercfg-b79ls\"" Nov 02 06:51:45.823061 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.823029 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-token\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.823246 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.823068 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istiod-ca-cert\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.823246 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.823176 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rgbp\" (UniqueName: \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-kube-api-access-9rgbp\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.823246 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.823217 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-credential-socket\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.823246 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.823242 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-data\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.823477 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.823389 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-socket\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.823477 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.823437 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-podinfo\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.823588 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.823514 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-certs\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.823641 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.823612 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-envoy\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.838912 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.838883 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr"] Nov 02 06:51:45.871979 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.871949 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:45.938627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.932065 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-envoy\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.938627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.932309 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-token\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.938627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.932349 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istiod-ca-cert\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.938627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.932392 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rgbp\" (UniqueName: \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-kube-api-access-9rgbp\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.938627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.932437 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-credential-socket\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.938627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.932472 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-data\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.938627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.932543 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-socket\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.938627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.932602 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-podinfo\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.938627 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.932635 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-certs\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.963099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.957659 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-certs\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.963099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.957985 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-credential-socket\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.963099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.959113 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istiod-ca-cert\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.963099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.959569 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-socket\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.963099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.959797 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-data\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:45.985503 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:45.985394 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-envoy\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:46.012738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:46.012698 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-podinfo\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:46.036099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:46.032834 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-token\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:46.051105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:46.051055 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rgbp\" (UniqueName: \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-kube-api-access-9rgbp\") pod \"e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:46.349878 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:46.349792 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:46.491903 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:46.491837 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:46.776193 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:46.776165 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr"] Nov 02 06:51:46.779456 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:46.779410 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:51:46.779583 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:46.779550 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:51:46.779692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:46.779639 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:51:47.720143 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:47.720092 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" event={"ID":"d0c0d533-68c3-4b4c-9769-fa8a193cba17","Type":"ContainerStarted","Data":"18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4"} Nov 02 06:51:47.720607 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:47.720160 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" event={"ID":"d0c0d533-68c3-4b4c-9769-fa8a193cba17","Type":"ContainerStarted","Data":"c00fbd38cdea6624fc29560988b73e6526719ac7be5272bdd93a29ed7884c204"} Nov 02 06:51:47.926838 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:47.926784 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" podStartSLOduration=2.926769459 podStartE2EDuration="2.926769459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:51:47.892799327 +0000 UTC m=+2084.341176103" watchObservedRunningTime="2026-11-02 06:51:47.926769459 +0000 UTC m=+2084.375146213" Nov 02 06:51:48.350108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:48.350044 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:48.356250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:48.356225 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:48.727501 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:48.727473 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:48.728349 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:48.728329 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:51:49.307926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:49.307888 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:49.922093 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:49.922045 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r"] Nov 02 06:51:50.219842 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:50.219753 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:50.456070 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:50.456043 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:50.773951 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:50.773918 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:50.985549 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:50.985517 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:53.705216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:53.705184 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:54.922526 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:54.922488 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" podUID="ea75ec12-0c13-4be6-ad9d-7cfa0f176780" containerName="payload-pre-processing" containerID="cri-o://f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352" gracePeriod=25 Nov 02 06:51:54.935866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:54.935841 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:55.324624 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.324526 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:55.373236 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.373207 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:51:55.462488 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.462452 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-kube-api-access-zlhx6\" (UniqueName: \"kubernetes.io/projected/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-kube-api-access-zlhx6\") pod \"ea75ec12-0c13-4be6-ad9d-7cfa0f176780\" (UID: \"ea75ec12-0c13-4be6-ad9d-7cfa0f176780\") " Nov 02 06:51:55.462688 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.462525 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-plugins-config-volume\") pod \"ea75ec12-0c13-4be6-ad9d-7cfa0f176780\" (UID: \"ea75ec12-0c13-4be6-ad9d-7cfa0f176780\") " Nov 02 06:51:55.462941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.462918 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-plugins-config-volume" pod "ea75ec12-0c13-4be6-ad9d-7cfa0f176780" (UID: "ea75ec12-0c13-4be6-ad9d-7cfa0f176780"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:51:55.464526 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.464506 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-kube-api-access-zlhx6" pod "ea75ec12-0c13-4be6-ad9d-7cfa0f176780" (UID: "ea75ec12-0c13-4be6-ad9d-7cfa0f176780"). InnerVolumeSpecName "kube-api-access-zlhx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:51:55.563870 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.563839 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:55.563870 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.563865 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlhx6\" (UniqueName: \"kubernetes.io/projected/ea75ec12-0c13-4be6-ad9d-7cfa0f176780-kube-api-access-zlhx6\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:51:55.607004 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.606925 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:55.862245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.858274 5025 generic.go:356] "Generic (PLEG): container finished" podID="ea75ec12-0c13-4be6-ad9d-7cfa0f176780" containerID="f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352" exitCode=0 Nov 02 06:51:55.862245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.858428 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" event={"ID":"ea75ec12-0c13-4be6-ad9d-7cfa0f176780","Type":"ContainerDied","Data":"f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352"} Nov 02 06:51:55.862245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.858478 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" event={"ID":"ea75ec12-0c13-4be6-ad9d-7cfa0f176780","Type":"ContainerDied","Data":"77635f51ba3157315eea3a58528a18730e5402b2bdf27c71ecd425df7bba6733"} Nov 02 06:51:55.862245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.858507 5025 scope.go:122] "RemoveContainer" containerID="f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352" Nov 02 06:51:55.862245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.858670 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r" Nov 02 06:51:55.901060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.901021 5025 scope.go:122] "RemoveContainer" containerID="f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352" Nov 02 06:51:55.902358 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:51:55.902299 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352\": container with ID starting with f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352 not found: ID does not exist" containerID="f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352" Nov 02 06:51:55.902581 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:55.902519 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352"} err="failed to get container status \"f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352\": rpc error: code = NotFound desc = could not find container \"f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352\": container with ID starting with f4dd6f1ceeb141502557c8474aa46be9a03f6fa7184a4a747ebad16ae1f2e352 not found: ID does not exist" Nov 02 06:51:56.017094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:56.017049 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r"] Nov 02 06:51:56.064195 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:56.064159 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-37b53e-66cfc478c5-dbx4r"] Nov 02 06:51:56.218274 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:56.218239 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:56.544971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:56.544888 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:56.730097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:56.729058 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:56.916294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:56.916264 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:57.417950 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:57.417920 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:57.705266 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:57.705187 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea75ec12-0c13-4be6-ad9d-7cfa0f176780" path="/var/lib/kubelet/pods/ea75ec12-0c13-4be6-ad9d-7cfa0f176780/volumes" Nov 02 06:51:57.729134 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:57.729040 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:57.997718 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:57.997487 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:51:58.244470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:51:58.236594 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:00.838393 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:00.838365 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k"] Nov 02 06:52:00.838826 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:00.838570 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" podUID="6602a158-b770-4ce3-80d2-6c050ad2c591" containerName="maas-api" containerID="cri-o://add4547793030e3b18adb9b109c6626c85150a0f5e9f2abc840ed140903b30f6" gracePeriod=30 Nov 02 06:52:00.981571 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:00.981531 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:01.084043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.083857 5025 generic.go:356] "Generic (PLEG): container finished" podID="6602a158-b770-4ce3-80d2-6c050ad2c591" containerID="add4547793030e3b18adb9b109c6626c85150a0f5e9f2abc840ed140903b30f6" exitCode=0 Nov 02 06:52:01.084043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.083936 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" event={"ID":"6602a158-b770-4ce3-80d2-6c050ad2c591","Type":"ContainerDied","Data":"add4547793030e3b18adb9b109c6626c85150a0f5e9f2abc840ed140903b30f6"} Nov 02 06:52:01.178174 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.178145 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:52:01.319156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.319126 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6602a158-b770-4ce3-80d2-6c050ad2c591-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6602a158-b770-4ce3-80d2-6c050ad2c591-openshift-service-ca\") pod \"6602a158-b770-4ce3-80d2-6c050ad2c591\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " Nov 02 06:52:01.319333 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.319216 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-metrics-certs\") pod \"6602a158-b770-4ce3-80d2-6c050ad2c591\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " Nov 02 06:52:01.319380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.319348 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6602a158-b770-4ce3-80d2-6c050ad2c591-kube-api-access-mlbxp\" (UniqueName: \"kubernetes.io/projected/6602a158-b770-4ce3-80d2-6c050ad2c591-kube-api-access-mlbxp\") pod \"6602a158-b770-4ce3-80d2-6c050ad2c591\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " Nov 02 06:52:01.319423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.319397 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-maas-api-tls\") pod \"6602a158-b770-4ce3-80d2-6c050ad2c591\" (UID: \"6602a158-b770-4ce3-80d2-6c050ad2c591\") " Nov 02 06:52:01.319588 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.319560 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6602a158-b770-4ce3-80d2-6c050ad2c591-openshift-service-ca" pod "6602a158-b770-4ce3-80d2-6c050ad2c591" (UID: "6602a158-b770-4ce3-80d2-6c050ad2c591"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:01.319978 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.319955 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6602a158-b770-4ce3-80d2-6c050ad2c591-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:01.321398 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.321370 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6602a158-b770-4ce3-80d2-6c050ad2c591-kube-api-access-mlbxp" pod "6602a158-b770-4ce3-80d2-6c050ad2c591" (UID: "6602a158-b770-4ce3-80d2-6c050ad2c591"). InnerVolumeSpecName "kube-api-access-mlbxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:01.321721 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.321693 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-metrics-certs" pod "6602a158-b770-4ce3-80d2-6c050ad2c591" (UID: "6602a158-b770-4ce3-80d2-6c050ad2c591"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:01.321721 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.321710 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-maas-api-tls" pod "6602a158-b770-4ce3-80d2-6c050ad2c591" (UID: "6602a158-b770-4ce3-80d2-6c050ad2c591"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:01.424464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.424374 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:01.424464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.424415 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlbxp\" (UniqueName: \"kubernetes.io/projected/6602a158-b770-4ce3-80d2-6c050ad2c591-kube-api-access-mlbxp\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:01.424464 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:01.424429 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6602a158-b770-4ce3-80d2-6c050ad2c591-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:02.091048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:02.090963 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" Nov 02 06:52:02.091048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:02.090990 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k" event={"ID":"6602a158-b770-4ce3-80d2-6c050ad2c591","Type":"ContainerDied","Data":"9ea800a6f0e176b46c67e186e5a1bb639d7b7e414d75f89aa2662babf25dff23"} Nov 02 06:52:02.091530 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:02.091052 5025 scope.go:122] "RemoveContainer" containerID="add4547793030e3b18adb9b109c6626c85150a0f5e9f2abc840ed140903b30f6" Nov 02 06:52:02.101517 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:02.101496 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:02.252702 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:02.252660 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k"] Nov 02 06:52:02.299836 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:02.299802 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-70898f-d9476cfcf-8z25k"] Nov 02 06:52:02.539453 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:02.539418 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:02.707021 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:52:02.706987 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:52:02.776097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:02.775165 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:03.300557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:03.300529 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:03.486170 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:03.486139 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:03.684282 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:03.684171 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:03.843099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:03.841636 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6602a158-b770-4ce3-80d2-6c050ad2c591" path="/var/lib/kubelet/pods/6602a158-b770-4ce3-80d2-6c050ad2c591/volumes" Nov 02 06:52:03.880341 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:03.880304 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:04.107989 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:04.107902 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:04.314302 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:04.314253 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:04.518005 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:04.517978 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v"] Nov 02 06:52:04.518247 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:04.518226 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" containerID="cri-o://a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407" gracePeriod=30 Nov 02 06:52:04.564784 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:04.564757 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:04.773355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:04.773257 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:05.346260 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:05.346225 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:06.041160 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:06.000739 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:06.533954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:06.533916 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:08.992122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:08.992090 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq"] Nov 02 06:52:09.147132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.147059 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:09.581556 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.575577 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:09.684998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.684968 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t"] Nov 02 06:52:09.778965 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.778865 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:09.804371 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.804312 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:52:09.823548 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.823523 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-credential-socket\") pod \"c68a4b47-40b0-493f-a07e-721871e27a0f\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " Nov 02 06:52:09.823664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.823567 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-token\" (UniqueName: \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-token\") pod \"c68a4b47-40b0-493f-a07e-721871e27a0f\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " Nov 02 06:52:09.823664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.823648 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c68a4b47-40b0-493f-a07e-721871e27a0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68a4b47-40b0-493f-a07e-721871e27a0f-istiod-ca-cert\") pod \"c68a4b47-40b0-493f-a07e-721871e27a0f\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " Nov 02 06:52:09.823783 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.823668 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-socket\") pod \"c68a4b47-40b0-493f-a07e-721871e27a0f\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " Nov 02 06:52:09.823783 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.823698 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-kube-api-access-66lh9\" (UniqueName: \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-kube-api-access-66lh9\") pod \"c68a4b47-40b0-493f-a07e-721871e27a0f\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " Nov 02 06:52:09.823783 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.823730 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-data\") pod \"c68a4b47-40b0-493f-a07e-721871e27a0f\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " Nov 02 06:52:09.823783 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.823758 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-certs\") pod \"c68a4b47-40b0-493f-a07e-721871e27a0f\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " Nov 02 06:52:09.823783 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.823759 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-credential-socket" pod "c68a4b47-40b0-493f-a07e-721871e27a0f" (UID: "c68a4b47-40b0-493f-a07e-721871e27a0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:09.824042 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.823840 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-podinfo\") pod \"c68a4b47-40b0-493f-a07e-721871e27a0f\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " Nov 02 06:52:09.824042 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.823876 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-envoy\") pod \"c68a4b47-40b0-493f-a07e-721871e27a0f\" (UID: \"c68a4b47-40b0-493f-a07e-721871e27a0f\") " Nov 02 06:52:09.824169 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.824070 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-certs" pod "c68a4b47-40b0-493f-a07e-721871e27a0f" (UID: "c68a4b47-40b0-493f-a07e-721871e27a0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:09.824231 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.824157 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c68a4b47-40b0-493f-a07e-721871e27a0f-istiod-ca-cert" pod "c68a4b47-40b0-493f-a07e-721871e27a0f" (UID: "c68a4b47-40b0-493f-a07e-721871e27a0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:09.824280 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.824235 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-data" pod "c68a4b47-40b0-493f-a07e-721871e27a0f" (UID: "c68a4b47-40b0-493f-a07e-721871e27a0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:09.824539 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.824519 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.824611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.824544 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c68a4b47-40b0-493f-a07e-721871e27a0f-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.824611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.824560 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.824611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.824582 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.824704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.824657 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-socket" pod "c68a4b47-40b0-493f-a07e-721871e27a0f" (UID: "c68a4b47-40b0-493f-a07e-721871e27a0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:09.825937 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.825915 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-envoy" pod "c68a4b47-40b0-493f-a07e-721871e27a0f" (UID: "c68a4b47-40b0-493f-a07e-721871e27a0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:09.825937 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.825915 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-token" pod "c68a4b47-40b0-493f-a07e-721871e27a0f" (UID: "c68a4b47-40b0-493f-a07e-721871e27a0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:09.826330 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.826307 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-podinfo" pod "c68a4b47-40b0-493f-a07e-721871e27a0f" (UID: "c68a4b47-40b0-493f-a07e-721871e27a0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:52:09.826330 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.826319 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-kube-api-access-66lh9" pod "c68a4b47-40b0-493f-a07e-721871e27a0f" (UID: "c68a4b47-40b0-493f-a07e-721871e27a0f"). InnerVolumeSpecName "kube-api-access-66lh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:09.926388 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.926350 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.926388 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.926382 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.926388 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.926394 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.926621 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.926406 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c68a4b47-40b0-493f-a07e-721871e27a0f-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.926621 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.926418 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66lh9\" (UniqueName: \"kubernetes.io/projected/c68a4b47-40b0-493f-a07e-721871e27a0f-kube-api-access-66lh9\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:09.977625 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:09.977588 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:10.173705 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.173662 5025 generic.go:356] "Generic (PLEG): container finished" podID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerID="a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407" exitCode=0 Nov 02 06:52:10.179259 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.173707 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" Nov 02 06:52:10.181470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.173723 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" event={"ID":"c68a4b47-40b0-493f-a07e-721871e27a0f","Type":"ContainerDied","Data":"a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407"} Nov 02 06:52:10.181470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.179739 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v" event={"ID":"c68a4b47-40b0-493f-a07e-721871e27a0f","Type":"ContainerDied","Data":"ceea010347181beafc3d84b674c4bf06ee8d7ee0dfaf0bf5f966fc3c119c7653"} Nov 02 06:52:10.181470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.179793 5025 scope.go:122] "RemoveContainer" containerID="a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407" Nov 02 06:52:10.211918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.207375 5025 scope.go:122] "RemoveContainer" containerID="356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8" Nov 02 06:52:10.219001 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.218952 5025 scope.go:122] "RemoveContainer" containerID="a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407" Nov 02 06:52:10.220271 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:52:10.220236 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407\": container with ID starting with a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407 not found: ID does not exist" containerID="a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407" Nov 02 06:52:10.220408 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.220282 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407"} err="failed to get container status \"a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407\": rpc error: code = NotFound desc = could not find container \"a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407\": container with ID starting with a2aef3eed581396b9adbed1b97e99fc359beeb472a495e23375ecfb34f430407 not found: ID does not exist" Nov 02 06:52:10.220408 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.220308 5025 scope.go:122] "RemoveContainer" containerID="356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8" Nov 02 06:52:10.220667 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:52:10.220639 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8\": container with ID starting with 356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8 not found: ID does not exist" containerID="356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8" Nov 02 06:52:10.220779 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.220675 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8"} err="failed to get container status \"356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8\": rpc error: code = NotFound desc = could not find container \"356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8\": container with ID starting with 356f6527efb9ff9e35a60eee471c26ad2c50eacf0b5171bb27b92980b87e94e8 not found: ID does not exist" Nov 02 06:52:10.249957 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.249914 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:10.346432 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.346391 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66"] Nov 02 06:52:10.346653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.346628 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" containerID="cri-o://3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65" gracePeriod=30 Nov 02 06:52:10.354119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.354089 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v"] Nov 02 06:52:10.390272 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.390244 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-6d13c3-openshift-default-5dd5bd7d97-wwc2v"] Nov 02 06:52:10.489307 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:10.473565 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:11.669810 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:11.669777 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" path="/var/lib/kubelet/pods/c68a4b47-40b0-493f-a07e-721871e27a0f/volumes" Nov 02 06:52:13.502801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:13.502771 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:13.827922 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:13.827793 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:14.932108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:14.928853 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:15.649187 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.649161 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:52:15.714940 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.714907 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-data\") pod \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " Nov 02 06:52:15.715142 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.714993 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-envoy\") pod \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " Nov 02 06:52:15.715142 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715028 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istiod-ca-cert\") pod \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " Nov 02 06:52:15.715142 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715132 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-credential-socket\") pod \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " Nov 02 06:52:15.715396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715152 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-podinfo\") pod \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " Nov 02 06:52:15.715396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715186 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-data" pod "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" (UID: "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:15.715396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715203 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-certs\") pod \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " Nov 02 06:52:15.715396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715281 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-kube-api-access-4qjbw\" (UniqueName: \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-kube-api-access-4qjbw\") pod \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " Nov 02 06:52:15.715396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715389 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-token\" (UniqueName: \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-token\") pod \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " Nov 02 06:52:15.715664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715423 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-credential-socket" pod "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" (UID: "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:15.715664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715445 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-socket\") pod \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\" (UID: \"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7\") " Nov 02 06:52:15.715664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715579 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istiod-ca-cert" pod "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" (UID: "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:15.715835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715782 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-socket" pod "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" (UID: "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:15.715835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.715569 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-certs" pod "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" (UID: "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:15.716104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.716037 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:15.716104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.716065 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:15.716104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.716096 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:15.716304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.716109 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:15.716304 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.716123 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:15.717401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.717369 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-envoy" pod "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" (UID: "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:15.717499 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.717411 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-podinfo" pod "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" (UID: "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:52:15.717551 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.717489 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-kube-api-access-4qjbw" pod "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" (UID: "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7"). InnerVolumeSpecName "kube-api-access-4qjbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:15.717551 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.717496 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-token" pod "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" (UID: "2a4894f3-c78f-4acb-81b3-09dc3abcb0f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:15.817675 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.817644 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qjbw\" (UniqueName: \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-kube-api-access-4qjbw\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:15.817830 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.817693 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:15.817830 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.817703 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:15.817830 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:15.817713 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:16.239267 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.239231 5025 generic.go:356] "Generic (PLEG): container finished" podID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerID="3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65" exitCode=0 Nov 02 06:52:16.239760 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.239300 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" Nov 02 06:52:16.239760 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.239314 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" event={"ID":"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7","Type":"ContainerDied","Data":"3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65"} Nov 02 06:52:16.239760 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.239386 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66" event={"ID":"2a4894f3-c78f-4acb-81b3-09dc3abcb0f7","Type":"ContainerDied","Data":"eb5aee44d4dc3d761dcd75fa4ac2e2ea0e002a486c1c11e09fdd06b6fa7ba1cf"} Nov 02 06:52:16.239760 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.239419 5025 scope.go:122] "RemoveContainer" containerID="3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65" Nov 02 06:52:16.251163 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.251145 5025 scope.go:122] "RemoveContainer" containerID="3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879" Nov 02 06:52:16.260058 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.260040 5025 scope.go:122] "RemoveContainer" containerID="3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65" Nov 02 06:52:16.260370 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:52:16.260350 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65\": container with ID starting with 3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65 not found: ID does not exist" containerID="3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65" Nov 02 06:52:16.260454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.260384 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65"} err="failed to get container status \"3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65\": rpc error: code = NotFound desc = could not find container \"3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65\": container with ID starting with 3bc3321f0de822ab1605aef2340d164dec108f76c09be11a641cc1b90bb7fe65 not found: ID does not exist" Nov 02 06:52:16.260454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.260409 5025 scope.go:122] "RemoveContainer" containerID="3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879" Nov 02 06:52:16.260634 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:52:16.260614 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879\": container with ID starting with 3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879 not found: ID does not exist" containerID="3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879" Nov 02 06:52:16.260676 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.260640 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879"} err="failed to get container status \"3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879\": rpc error: code = NotFound desc = could not find container \"3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879\": container with ID starting with 3400482b7bf3e7a28f58090af9e7ca10b88e86f2da50334e990904654a237879 not found: ID does not exist" Nov 02 06:52:16.397212 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.397179 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66"] Nov 02 06:52:16.444317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:16.444280 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-70898f-openshift-default-54fc8d94f7-hvg66"] Nov 02 06:52:17.093675 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:17.093639 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:17.288854 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:17.288824 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:17.669542 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:17.669510 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" path="/var/lib/kubelet/pods/2a4894f3-c78f-4acb-81b3-09dc3abcb0f7/volumes" Nov 02 06:52:18.125162 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:18.115666 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:19.885478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:19.885443 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:22.967337 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:22.967286 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:23.162867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:23.162833 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:24.034423 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.034380 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" containerID="cri-o://112541316a8a03b0992492c6051cd6959e921a844f7cb2d67f73299eba1d7fcb" gracePeriod=45 Nov 02 06:52:24.324786 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.324743 5025 generic.go:356] "Generic (PLEG): container finished" podID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerID="112541316a8a03b0992492c6051cd6959e921a844f7cb2d67f73299eba1d7fcb" exitCode=0 Nov 02 06:52:24.325192 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.325161 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" event={"ID":"7c6d6102-1694-4edc-baec-59156cdcf81e","Type":"ContainerDied","Data":"112541316a8a03b0992492c6051cd6959e921a844f7cb2d67f73299eba1d7fcb"} Nov 02 06:52:24.446146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.446058 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:52:24.576048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.575962 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6d6102-1694-4edc-baec-59156cdcf81e-tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6d6102-1694-4edc-baec-59156cdcf81e-tls-certs\") pod \"7c6d6102-1694-4edc-baec-59156cdcf81e\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " Nov 02 06:52:24.576048 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.576000 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-home\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-home\") pod \"7c6d6102-1694-4edc-baec-59156cdcf81e\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " Nov 02 06:52:24.576307 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.576288 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-model-cache\") pod \"7c6d6102-1694-4edc-baec-59156cdcf81e\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " Nov 02 06:52:24.576396 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.576360 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6d6102-1694-4edc-baec-59156cdcf81e-kube-api-access-nxkm2\" (UniqueName: \"kubernetes.io/projected/7c6d6102-1694-4edc-baec-59156cdcf81e-kube-api-access-nxkm2\") pod \"7c6d6102-1694-4edc-baec-59156cdcf81e\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " Nov 02 06:52:24.576454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.576385 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-home" pod "7c6d6102-1694-4edc-baec-59156cdcf81e" (UID: "7c6d6102-1694-4edc-baec-59156cdcf81e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.576454 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.576431 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-tmp-dir\") pod \"7c6d6102-1694-4edc-baec-59156cdcf81e\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " Nov 02 06:52:24.576558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.576462 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-dshm\") pod \"7c6d6102-1694-4edc-baec-59156cdcf81e\" (UID: \"7c6d6102-1694-4edc-baec-59156cdcf81e\") " Nov 02 06:52:24.576648 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.576624 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-model-cache" pod "7c6d6102-1694-4edc-baec-59156cdcf81e" (UID: "7c6d6102-1694-4edc-baec-59156cdcf81e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.576948 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.576927 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-tmp-dir" pod "7c6d6102-1694-4edc-baec-59156cdcf81e" (UID: "7c6d6102-1694-4edc-baec-59156cdcf81e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.577276 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.577257 5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-model-cache\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.577397 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.577383 5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-tmp-dir\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.577500 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.577479 5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-home\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.578893 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.578866 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-dshm" pod "7c6d6102-1694-4edc-baec-59156cdcf81e" (UID: "7c6d6102-1694-4edc-baec-59156cdcf81e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:24.579774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.579742 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6d6102-1694-4edc-baec-59156cdcf81e-kube-api-access-nxkm2" pod "7c6d6102-1694-4edc-baec-59156cdcf81e" (UID: "7c6d6102-1694-4edc-baec-59156cdcf81e"). InnerVolumeSpecName "kube-api-access-nxkm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:24.580506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.580480 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6d6102-1694-4edc-baec-59156cdcf81e-tls-certs" pod "7c6d6102-1694-4edc-baec-59156cdcf81e" (UID: "7c6d6102-1694-4edc-baec-59156cdcf81e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:24.679164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.679125 5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c6d6102-1694-4edc-baec-59156cdcf81e-tls-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.679164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.679163 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxkm2\" (UniqueName: \"kubernetes.io/projected/7c6d6102-1694-4edc-baec-59156cdcf81e-kube-api-access-nxkm2\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.679413 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.679177 5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c6d6102-1694-4edc-baec-59156cdcf81e-dshm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:24.712363 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.712314 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" podUID="4878a04e-ddb9-4772-a068-3cdba2e83d5c" containerName="main" containerID="cri-o://e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73" gracePeriod=45 Nov 02 06:52:24.883105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:24.873774 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:25.035217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.035185 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:52:25.084242 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.084216 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tls-certs\") pod \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " Nov 02 06:52:25.084384 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.084289 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-model-cache\") pod \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " Nov 02 06:52:25.084384 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.084329 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-dshm\") pod \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " Nov 02 06:52:25.084518 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.084476 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tmp-dir\") pod \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " Nov 02 06:52:25.084584 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.084521 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-home\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-home\") pod \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " Nov 02 06:52:25.084584 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.084557 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4878a04e-ddb9-4772-a068-3cdba2e83d5c-kube-api-access-5s9q6\" (UniqueName: \"kubernetes.io/projected/4878a04e-ddb9-4772-a068-3cdba2e83d5c-kube-api-access-5s9q6\") pod \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\" (UID: \"4878a04e-ddb9-4772-a068-3cdba2e83d5c\") " Nov 02 06:52:25.084697 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.084595 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-model-cache" pod "4878a04e-ddb9-4772-a068-3cdba2e83d5c" (UID: "4878a04e-ddb9-4772-a068-3cdba2e83d5c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:25.084807 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.084776 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tmp-dir" pod "4878a04e-ddb9-4772-a068-3cdba2e83d5c" (UID: "4878a04e-ddb9-4772-a068-3cdba2e83d5c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:25.084872 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.084825 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-home" pod "4878a04e-ddb9-4772-a068-3cdba2e83d5c" (UID: "4878a04e-ddb9-4772-a068-3cdba2e83d5c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:25.085294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.085272 5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-model-cache\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:25.085294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.085295 5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tmp-dir\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:25.085469 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.085308 5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-home\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:25.086628 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.086607 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tls-certs" pod "4878a04e-ddb9-4772-a068-3cdba2e83d5c" (UID: "4878a04e-ddb9-4772-a068-3cdba2e83d5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:52:25.086628 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.086617 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-dshm" pod "4878a04e-ddb9-4772-a068-3cdba2e83d5c" (UID: "4878a04e-ddb9-4772-a068-3cdba2e83d5c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:25.086918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.086904 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4878a04e-ddb9-4772-a068-3cdba2e83d5c-kube-api-access-5s9q6" pod "4878a04e-ddb9-4772-a068-3cdba2e83d5c" (UID: "4878a04e-ddb9-4772-a068-3cdba2e83d5c"). InnerVolumeSpecName "kube-api-access-5s9q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:25.187186 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.187147 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s9q6\" (UniqueName: \"kubernetes.io/projected/4878a04e-ddb9-4772-a068-3cdba2e83d5c-kube-api-access-5s9q6\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:25.187186 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.187188 5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4878a04e-ddb9-4772-a068-3cdba2e83d5c-tls-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:25.187433 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.187203 5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4878a04e-ddb9-4772-a068-3cdba2e83d5c-dshm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:25.332931 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.332889 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" Nov 02 06:52:25.333157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.332930 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq" event={"ID":"7c6d6102-1694-4edc-baec-59156cdcf81e","Type":"ContainerDied","Data":"36e11a44180272787d73cf43f28238057a06a5f49eb3c763fcbba24f8265bc2d"} Nov 02 06:52:25.333157 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.332986 5025 scope.go:122] "RemoveContainer" containerID="112541316a8a03b0992492c6051cd6959e921a844f7cb2d67f73299eba1d7fcb" Nov 02 06:52:25.334741 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.334713 5025 generic.go:356] "Generic (PLEG): container finished" podID="4878a04e-ddb9-4772-a068-3cdba2e83d5c" containerID="e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73" exitCode=0 Nov 02 06:52:25.334877 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.334755 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" Nov 02 06:52:25.334877 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.334791 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" event={"ID":"4878a04e-ddb9-4772-a068-3cdba2e83d5c","Type":"ContainerDied","Data":"e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73"} Nov 02 06:52:25.334877 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.334820 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t" event={"ID":"4878a04e-ddb9-4772-a068-3cdba2e83d5c","Type":"ContainerDied","Data":"9e74d6d73dce8c69c10fbc564411cb7c2494a0ddc581c41eda45e6b2335baf4c"} Nov 02 06:52:25.344417 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.344397 5025 scope.go:122] "RemoveContainer" containerID="e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73" Nov 02 06:52:25.353030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.353011 5025 scope.go:122] "RemoveContainer" containerID="e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73" Nov 02 06:52:25.353274 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:52:25.353258 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73\": container with ID starting with e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73 not found: ID does not exist" containerID="e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73" Nov 02 06:52:25.353331 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.353281 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73"} err="failed to get container status \"e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73\": rpc error: code = NotFound desc = could not find container \"e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73\": container with ID starting with e5146f1265fc4df2af933b6ed906032752e8e16474e3c2ac0041633cdcca9d73 not found: ID does not exist" Nov 02 06:52:25.492764 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.492732 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq"] Nov 02 06:52:25.496859 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.496831 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-bdcd80/distinct-2-w0-bdcd80-kserve-7d8986fdc5-mz7pq"] Nov 02 06:52:25.552339 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.552316 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:25.601119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.601063 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t"] Nov 02 06:52:25.604926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.604898 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-3bf02ebf/e2e-lifecycle-model-3bf02ebf-kserve-57b44fff69-ks77t"] Nov 02 06:52:25.670429 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.670400 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4878a04e-ddb9-4772-a068-3cdba2e83d5c" path="/var/lib/kubelet/pods/4878a04e-ddb9-4772-a068-3cdba2e83d5c/volumes" Nov 02 06:52:25.670878 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:25.670865 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" path="/var/lib/kubelet/pods/7c6d6102-1694-4edc-baec-59156cdcf81e/volumes" Nov 02 06:52:29.935605 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:29.935567 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:31.878645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:31.878613 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:32.707284 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:52:32.707245 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:52:35.007134 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:35.001933 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:39.752099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:39.748249 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:43.970506 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:43.970466 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:45.459299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:45.459266 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:45.688237 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:45.688203 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:45.954664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:45.954622 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:46.366112 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:46.366028 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:46.618535 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:46.618504 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:46.912202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:46.912070 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:47.175841 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:47.175810 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:47.478675 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:47.478590 5025 generic.go:356] "Generic (PLEG): container finished" podID="36972b30-f157-4a06-aa99-55a5e941643c" containerID="96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c" exitCode=0 Nov 02 06:52:47.478835 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:47.478665 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" event={"ID":"36972b30-f157-4a06-aa99-55a5e941643c","Type":"ContainerDied","Data":"96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c"} Nov 02 06:52:47.479168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:47.479149 5025 scope.go:122] "RemoveContainer" containerID="96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c" Nov 02 06:52:47.480992 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:47.480957 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:52:47.481093 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:47.481044 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:52:47.481138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:47.481119 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:52:47.684907 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:47.684472 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:48.065326 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:48.065281 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:48.510754 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:48.510709 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" event={"ID":"36972b30-f157-4a06-aa99-55a5e941643c","Type":"ContainerStarted","Data":"5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b"} Nov 02 06:52:48.776244 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:48.776167 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:52:48.777367 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:48.777344 5025 patch_prober.go:29] interesting pod/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 06:52:48.777489 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:48.777387 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 06:52:49.042258 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:49.042169 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:49.777030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:49.776996 5025 patch_prober.go:29] interesting pod/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 06:52:49.777445 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:49.777050 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 06:52:50.624664 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:50.624631 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:52:50.753018 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:50.752985 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:50.776964 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:50.776931 5025 patch_prober.go:29] interesting pod/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 06:52:50.777138 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:50.776992 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 06:52:51.776902 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:51.776862 5025 patch_prober.go:29] interesting pod/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 06:52:51.777107 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:51.776919 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 06:52:52.551859 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:52.551827 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:52.741738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:52.741705 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:52.776827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:52.776787 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj"] Nov 02 06:52:52.777013 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:52.776866 5025 patch_prober.go:29] interesting pod/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body= Nov 02 06:52:52.777013 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:52.776911 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" Nov 02 06:52:52.777174 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:52.777030 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" containerID="cri-o://517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53" gracePeriod=30 Nov 02 06:52:52.969676 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:52.969634 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:53.409769 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:53.409659 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:53.780880 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:53.780845 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:52:53.781939 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:53.781917 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 06:52:54.894346 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:54.894310 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:55.193230 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:55.193198 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:55.763409 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:55.763365 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:55.949460 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:55.949427 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:56.055114 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:56.055020 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr"] Nov 02 06:52:56.055260 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:56.055230 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" podUID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" containerName="istio-proxy" containerID="cri-o://fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7" gracePeriod=30 Nov 02 06:52:56.079310 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:56.079272 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8"] Nov 02 06:52:56.080384 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:56.080352 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" podUID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" containerName="istio-proxy" containerID="cri-o://628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d" gracePeriod=30 Nov 02 06:52:56.147973 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:56.147927 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:58.032756 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.032736 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:52:58.197998 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.197956 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istiod-ca-cert\") pod \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " Nov 02 06:52:58.198207 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.198053 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-credential-socket\") pod \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " Nov 02 06:52:58.198207 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.198178 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-kube-api-access-ttv9z\" (UniqueName: \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-kube-api-access-ttv9z\") pod \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " Nov 02 06:52:58.198354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.198208 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-envoy\") pod \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " Nov 02 06:52:58.198354 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.198335 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-socket\") pod \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " Nov 02 06:52:58.198460 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.198396 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-certs\") pod \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " Nov 02 06:52:58.198460 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.198450 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istiod-ca-cert" pod "bc2f04e7-114a-4ab0-8e5f-1035460391fd" (UID: "bc2f04e7-114a-4ab0-8e5f-1035460391fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:52:58.198563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.198468 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-token\" (UniqueName: \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-token\") pod \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " Nov 02 06:52:58.198563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.198511 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-podinfo\") pod \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " Nov 02 06:52:58.198563 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.198543 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-data\") pod \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\" (UID: \"bc2f04e7-114a-4ab0-8e5f-1035460391fd\") " Nov 02 06:52:58.198719 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.198700 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-socket" pod "bc2f04e7-114a-4ab0-8e5f-1035460391fd" (UID: "bc2f04e7-114a-4ab0-8e5f-1035460391fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:58.199203 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.199181 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:58.199397 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.199380 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:58.199541 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.199262 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-certs" pod "bc2f04e7-114a-4ab0-8e5f-1035460391fd" (UID: "bc2f04e7-114a-4ab0-8e5f-1035460391fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:58.199663 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.199421 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-data" pod "bc2f04e7-114a-4ab0-8e5f-1035460391fd" (UID: "bc2f04e7-114a-4ab0-8e5f-1035460391fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:58.199774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.199677 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-credential-socket" pod "bc2f04e7-114a-4ab0-8e5f-1035460391fd" (UID: "bc2f04e7-114a-4ab0-8e5f-1035460391fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:58.200766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.200741 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-kube-api-access-ttv9z" pod "bc2f04e7-114a-4ab0-8e5f-1035460391fd" (UID: "bc2f04e7-114a-4ab0-8e5f-1035460391fd"). InnerVolumeSpecName "kube-api-access-ttv9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:58.200766 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.200759 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-envoy" pod "bc2f04e7-114a-4ab0-8e5f-1035460391fd" (UID: "bc2f04e7-114a-4ab0-8e5f-1035460391fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:52:58.200970 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.200956 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-podinfo" pod "bc2f04e7-114a-4ab0-8e5f-1035460391fd" (UID: "bc2f04e7-114a-4ab0-8e5f-1035460391fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:52:58.201526 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.201506 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-token" pod "bc2f04e7-114a-4ab0-8e5f-1035460391fd" (UID: "bc2f04e7-114a-4ab0-8e5f-1035460391fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:52:58.300692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.300658 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:58.300692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.300688 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:58.300692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.300698 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:58.300919 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.300706 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:58.300919 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.300715 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:58.300919 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.300724 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttv9z\" (UniqueName: \"kubernetes.io/projected/bc2f04e7-114a-4ab0-8e5f-1035460391fd-kube-api-access-ttv9z\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:58.300919 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.300733 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc2f04e7-114a-4ab0-8e5f-1035460391fd-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:52:58.516890 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.516811 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:52:58.576268 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.576231 5025 generic.go:356] "Generic (PLEG): container finished" podID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerID="517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53" exitCode=0 Nov 02 06:52:58.576443 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.576289 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" Nov 02 06:52:58.576443 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.576305 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" event={"ID":"bc2f04e7-114a-4ab0-8e5f-1035460391fd","Type":"ContainerDied","Data":"517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53"} Nov 02 06:52:58.576443 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.576365 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj" event={"ID":"bc2f04e7-114a-4ab0-8e5f-1035460391fd","Type":"ContainerDied","Data":"fc7141ae38dda2040bf466026a8997c1199b18e8ad613cedf14f008542896419"} Nov 02 06:52:58.576443 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.576387 5025 scope.go:122] "RemoveContainer" containerID="517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53" Nov 02 06:52:58.588127 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.588069 5025 scope.go:122] "RemoveContainer" containerID="56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f" Nov 02 06:52:58.597069 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.597051 5025 scope.go:122] "RemoveContainer" containerID="517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53" Nov 02 06:52:58.597363 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:52:58.597344 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53\": container with ID starting with 517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53 not found: ID does not exist" containerID="517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53" Nov 02 06:52:58.597443 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.597374 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53"} err="failed to get container status \"517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53\": rpc error: code = NotFound desc = could not find container \"517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53\": container with ID starting with 517501f173a17f5557863421b04419d427c7ab4f24c0211c7e49d22cbf074e53 not found: ID does not exist" Nov 02 06:52:58.597443 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.597397 5025 scope.go:122] "RemoveContainer" containerID="56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f" Nov 02 06:52:58.597676 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:52:58.597656 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f\": container with ID starting with 56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f not found: ID does not exist" containerID="56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f" Nov 02 06:52:58.597717 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.597683 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f"} err="failed to get container status \"56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f\": rpc error: code = NotFound desc = could not find container \"56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f\": container with ID starting with 56bbee83d75d5e91fd67f72d66960c00c6be2112936e1a868d1bffbd444dda7f not found: ID does not exist" Nov 02 06:52:58.744871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.744840 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj"] Nov 02 06:52:58.791647 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:58.791565 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-37b53e-openshift-default-64f4dd4f8f-2r4sj"] Nov 02 06:52:59.669457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:59.669429 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" path="/var/lib/kubelet/pods/bc2f04e7-114a-4ab0-8e5f-1035460391fd/volumes" Nov 02 06:52:59.914738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:52:59.908467 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:00.341776 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.341745 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:00.821189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821158 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269"] Nov 02 06:53:00.821738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821722 5025 cpu_manager.go:414] "containerMap: removing container" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" Nov 02 06:53:00.821738 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821738 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821752 5025 cpu_manager.go:414] "containerMap: removing container" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821757 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e296f80-34b3-44ad-b978-bcd9b99f19c1" containerName="istio-proxy" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821768 5025 cpu_manager.go:414] "containerMap: removing container" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821772 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821783 5025 cpu_manager.go:414] "containerMap: removing container" podUID="ea75ec12-0c13-4be6-ad9d-7cfa0f176780" containerName="payload-pre-processing" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821788 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea75ec12-0c13-4be6-ad9d-7cfa0f176780" containerName="payload-pre-processing" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821797 5025 cpu_manager.go:414] "containerMap: removing container" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821802 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821811 5025 cpu_manager.go:414] "containerMap: removing container" podUID="6602a158-b770-4ce3-80d2-6c050ad2c591" containerName="maas-api" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821815 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6602a158-b770-4ce3-80d2-6c050ad2c591" containerName="maas-api" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821840 5025 cpu_manager.go:414] "containerMap: removing container" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821845 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821854 5025 cpu_manager.go:414] "containerMap: removing container" podUID="4878a04e-ddb9-4772-a068-3cdba2e83d5c" containerName="main" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821858 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4878a04e-ddb9-4772-a068-3cdba2e83d5c" containerName="main" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821867 5025 cpu_manager.go:414] "containerMap: removing container" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" Nov 02 06:53:00.821864 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821871 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" Nov 02 06:53:00.822344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821948 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" Nov 02 06:53:00.822344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821955 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" Nov 02 06:53:00.822344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821961 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="4878a04e-ddb9-4772-a068-3cdba2e83d5c" containerName="main" Nov 02 06:53:00.822344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821967 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6d6102-1694-4edc-baec-59156cdcf81e" containerName="main" Nov 02 06:53:00.822344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821972 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" Nov 02 06:53:00.822344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821979 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="c68a4b47-40b0-493f-a07e-721871e27a0f" containerName="istio-proxy" Nov 02 06:53:00.822344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821985 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea75ec12-0c13-4be6-ad9d-7cfa0f176780" containerName="payload-pre-processing" Nov 02 06:53:00.822344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.821991 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="6602a158-b770-4ce3-80d2-6c050ad2c591" containerName="maas-api" Nov 02 06:53:00.825155 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.825136 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.848956 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.848927 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-data\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.849132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.848972 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63732c47-cea0-4015-b099-8b7b465ccdff-istiod-ca-cert\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.849132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.849055 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-envoy\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.849251 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.849193 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63732c47-cea0-4015-b099-8b7b465ccdff-istio-podinfo\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.849300 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.849282 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-certs\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.849344 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.849311 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-socket\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.849412 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.849380 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-credential-socket\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.849529 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.849515 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v6zt\" (UniqueName: \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-kube-api-access-6v6zt\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.849592 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.849537 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-istio-token\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.885069 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.885042 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2e9f81-openshift-default-dockercfg-jm8j4\"" Nov 02 06:53:00.892857 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.892824 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:00.930361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.930331 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269"] Nov 02 06:53:00.950515 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.950478 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63732c47-cea0-4015-b099-8b7b465ccdff-istio-podinfo\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.950667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.950626 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-certs\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.950667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.950660 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-socket\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.950787 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.950681 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-credential-socket\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.950787 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.950742 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v6zt\" (UniqueName: \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-kube-api-access-6v6zt\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.950787 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.950772 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-istio-token\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.950933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.950831 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-data\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.950933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.950869 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63732c47-cea0-4015-b099-8b7b465ccdff-istiod-ca-cert\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.950933 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.950911 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-envoy\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.951121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.951044 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-certs\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.951121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.951067 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-socket\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.951234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.951150 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-credential-socket\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.951270 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.951230 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-data\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.951628 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.951606 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63732c47-cea0-4015-b099-8b7b465ccdff-istiod-ca-cert\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.953051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.953031 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63732c47-cea0-4015-b099-8b7b465ccdff-istio-podinfo\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:00.953159 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:00.953059 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-envoy\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:01.009343 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.009313 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v6zt\" (UniqueName: \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-kube-api-access-6v6zt\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:01.009866 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.009849 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-istio-token\") pod \"e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:01.095455 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.095428 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:01.138914 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.138891 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:01.298931 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.298845 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:01.543557 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.543527 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:53:01.567845 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.567814 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-certs\") pod \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " Nov 02 06:53:01.568040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.567918 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-podinfo\") pod \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " Nov 02 06:53:01.568040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568018 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-data\") pod \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " Nov 02 06:53:01.568167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568070 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istiod-ca-cert\") pod \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " Nov 02 06:53:01.568167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568131 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-kube-api-access-cpg2t\" (UniqueName: \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-kube-api-access-cpg2t\") pod \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " Nov 02 06:53:01.568167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568150 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-certs" pod "896cc162-c5d2-45c0-a86a-026df2b1b6f9" (UID: "896cc162-c5d2-45c0-a86a-026df2b1b6f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.568167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568163 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-token\" (UniqueName: \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-token\") pod \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " Nov 02 06:53:01.568400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568218 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-socket\") pod \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " Nov 02 06:53:01.568400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568309 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-data" pod "896cc162-c5d2-45c0-a86a-026df2b1b6f9" (UID: "896cc162-c5d2-45c0-a86a-026df2b1b6f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.568400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568323 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-envoy\") pod \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " Nov 02 06:53:01.568400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568356 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-credential-socket\") pod \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\" (UID: \"896cc162-c5d2-45c0-a86a-026df2b1b6f9\") " Nov 02 06:53:01.568613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568541 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istiod-ca-cert" pod "896cc162-c5d2-45c0-a86a-026df2b1b6f9" (UID: "896cc162-c5d2-45c0-a86a-026df2b1b6f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:01.568914 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568843 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-credential-socket" pod "896cc162-c5d2-45c0-a86a-026df2b1b6f9" (UID: "896cc162-c5d2-45c0-a86a-026df2b1b6f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.568981 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.568940 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-socket" pod "896cc162-c5d2-45c0-a86a-026df2b1b6f9" (UID: "896cc162-c5d2-45c0-a86a-026df2b1b6f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.569171 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.569143 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.569171 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.569172 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.569361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.569211 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.569361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.569226 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.569361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.569240 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.570625 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.570597 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-token" pod "896cc162-c5d2-45c0-a86a-026df2b1b6f9" (UID: "896cc162-c5d2-45c0-a86a-026df2b1b6f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.570625 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.570620 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-kube-api-access-cpg2t" pod "896cc162-c5d2-45c0-a86a-026df2b1b6f9" (UID: "896cc162-c5d2-45c0-a86a-026df2b1b6f9"). InnerVolumeSpecName "kube-api-access-cpg2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.571268 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.571250 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-envoy" pod "896cc162-c5d2-45c0-a86a-026df2b1b6f9" (UID: "896cc162-c5d2-45c0-a86a-026df2b1b6f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.571346 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.571264 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-podinfo" pod "896cc162-c5d2-45c0-a86a-026df2b1b6f9" (UID: "896cc162-c5d2-45c0-a86a-026df2b1b6f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:53:01.591565 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.591543 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:53:01.598765 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.598740 5025 generic.go:356] "Generic (PLEG): container finished" podID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" containerID="628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d" exitCode=0 Nov 02 06:53:01.598869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.598818 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" Nov 02 06:53:01.598869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.598822 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" event={"ID":"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b","Type":"ContainerDied","Data":"628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d"} Nov 02 06:53:01.598869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.598862 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8" event={"ID":"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b","Type":"ContainerDied","Data":"4a61e758f8c9bc56a4f0e7f7167d08fb8742f260e0fc0f4fff4c855c3cb27b79"} Nov 02 06:53:01.599067 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.598883 5025 scope.go:122] "RemoveContainer" containerID="628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d" Nov 02 06:53:01.600651 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.600632 5025 generic.go:356] "Generic (PLEG): container finished" podID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" containerID="fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7" exitCode=0 Nov 02 06:53:01.600752 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.600673 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" Nov 02 06:53:01.600752 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.600716 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" event={"ID":"896cc162-c5d2-45c0-a86a-026df2b1b6f9","Type":"ContainerDied","Data":"fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7"} Nov 02 06:53:01.600857 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.600760 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr" event={"ID":"896cc162-c5d2-45c0-a86a-026df2b1b6f9","Type":"ContainerDied","Data":"88a9c817d557280313edb45c36085489d962a15224d479f428f99166853e682a"} Nov 02 06:53:01.611697 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.611676 5025 scope.go:122] "RemoveContainer" containerID="628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d" Nov 02 06:53:01.611990 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:53:01.611963 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d\": container with ID starting with 628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d not found: ID does not exist" containerID="628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d" Nov 02 06:53:01.612059 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.612003 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d"} err="failed to get container status \"628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d\": rpc error: code = NotFound desc = could not find container \"628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d\": container with ID starting with 628ed84391885ebba6c40c462989fff981c94da6f0349fd78bcd98c5141cf23d not found: ID does not exist" Nov 02 06:53:01.612059 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.612029 5025 scope.go:122] "RemoveContainer" containerID="fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7" Nov 02 06:53:01.622862 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.622842 5025 scope.go:122] "RemoveContainer" containerID="fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7" Nov 02 06:53:01.623210 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:53:01.623186 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7\": container with ID starting with fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7 not found: ID does not exist" containerID="fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7" Nov 02 06:53:01.623284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.623219 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7"} err="failed to get container status \"fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7\": rpc error: code = NotFound desc = could not find container \"fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7\": container with ID starting with fa73eed0d1a160575a23d423f92c408f0eb56e708c67d94af50dce24637380d7 not found: ID does not exist" Nov 02 06:53:01.669711 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.669679 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-kube-api-access-72gsm\" (UniqueName: \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-kube-api-access-72gsm\") pod \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " Nov 02 06:53:01.669890 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.669872 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-envoy\") pod \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " Nov 02 06:53:01.669962 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.669930 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-socket\") pod \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " Nov 02 06:53:01.670018 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.669989 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-token\" (UniqueName: \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-token\") pod \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " Nov 02 06:53:01.670069 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.670037 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-certs\") pod \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " Nov 02 06:53:01.670164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.670085 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-credential-socket\") pod \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " Nov 02 06:53:01.670218 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.670187 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-data\") pod \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " Nov 02 06:53:01.670275 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.670263 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istiod-ca-cert\") pod \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " Nov 02 06:53:01.670328 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.670298 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-podinfo\") pod \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\" (UID: \"2637f05a-9e43-47c5-9e2c-8bffb1e58e4b\") " Nov 02 06:53:01.670622 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.670595 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-certs" pod "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" (UID: "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.670782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.670763 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.670848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.670787 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpg2t\" (UniqueName: \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-kube-api-access-cpg2t\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.670848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.670802 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.670848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.670816 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.670848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.670830 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/896cc162-c5d2-45c0-a86a-026df2b1b6f9-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.671611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.671564 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istiod-ca-cert" pod "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" (UID: "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:01.671867 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.671841 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-credential-socket" pod "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" (UID: "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.671945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.671925 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-socket" pod "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" (UID: "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.672167 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.672143 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-data" pod "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" (UID: "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.672831 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.672806 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-envoy" pod "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" (UID: "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:01.673219 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.673194 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-podinfo" pod "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" (UID: "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:53:01.673960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.673929 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-token" pod "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" (UID: "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.674184 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.674161 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-kube-api-access-72gsm" pod "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" (UID: "2637f05a-9e43-47c5-9e2c-8bffb1e58e4b"). InnerVolumeSpecName "kube-api-access-72gsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:01.772250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.772173 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.772250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.772203 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72gsm\" (UniqueName: \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-kube-api-access-72gsm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.772250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.772212 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.772250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.772221 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.772250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.772229 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.772250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.772239 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.772250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.772246 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.772250 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.772254 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:01.811178 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.811146 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269"] Nov 02 06:53:01.812355 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:53:01.812327 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63732c47_cea0_4015_b099_8b7b465ccdff.slice/crio-07c1dbdb7b90a29b38975f689ac0e7ea90ff38c73f2276e2805ef43bdb0e6089 WatchSource:0}: Error finding container 07c1dbdb7b90a29b38975f689ac0e7ea90ff38c73f2276e2805ef43bdb0e6089: Status 404 returned error can't find the container with id 07c1dbdb7b90a29b38975f689ac0e7ea90ff38c73f2276e2805ef43bdb0e6089 Nov 02 06:53:01.815466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.815424 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:01.815549 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.815516 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:01.815585 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.815560 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:01.882218 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.882195 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr"] Nov 02 06:53:01.886588 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:01.886567 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-vbdqr"] Nov 02 06:53:02.020777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:02.020748 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8"] Nov 02 06:53:02.030140 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:02.027427 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3bf02ebf-openshift-default-6b7794758f-q7cd8"] Nov 02 06:53:02.610486 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:02.610456 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" event={"ID":"63732c47-cea0-4015-b099-8b7b465ccdff","Type":"ContainerStarted","Data":"7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2"} Nov 02 06:53:02.610632 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:02.610525 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" event={"ID":"63732c47-cea0-4015-b099-8b7b465ccdff","Type":"ContainerStarted","Data":"07c1dbdb7b90a29b38975f689ac0e7ea90ff38c73f2276e2805ef43bdb0e6089"} Nov 02 06:53:02.707530 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:53:02.707493 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:53:02.756527 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:02.756478 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" podStartSLOduration=2.756463794 podStartE2EDuration="2.756463794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:53:02.75571181 +0000 UTC m=+2159.204088565" watchObservedRunningTime="2026-11-02 06:53:02.756463794 +0000 UTC m=+2159.204840548" Nov 02 06:53:03.139313 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:03.139278 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:03.145233 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:03.145207 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:03.617868 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:03.617840 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:03.618853 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:03.618833 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:53:03.669925 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:03.669895 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" path="/var/lib/kubelet/pods/2637f05a-9e43-47c5-9e2c-8bffb1e58e4b/volumes" Nov 02 06:53:03.670526 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:03.670508 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" path="/var/lib/kubelet/pods/896cc162-c5d2-45c0-a86a-026df2b1b6f9/volumes" Nov 02 06:53:05.737264 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:05.737211 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:05.937617 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:05.937549 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:06.448655 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:06.448618 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:11.323822 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:11.323757 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:11.528336 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:11.528306 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:11.719739 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:11.719706 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:16.590666 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:16.590543 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:16.794729 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:16.793195 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:21.043600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.043560 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8"] Nov 02 06:53:21.044124 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.043796 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" podUID="b85be5a1-c949-4a43-9c96-a236c69e1b6c" containerName="maas-api" containerID="cri-o://9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e" gracePeriod=30 Nov 02 06:53:21.214114 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.214006 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:21.379894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.379872 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:53:21.451449 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.451418 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-metrics-certs\") pod \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " Nov 02 06:53:21.451595 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.451485 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85be5a1-c949-4a43-9c96-a236c69e1b6c-kube-api-access-rdmv8\" (UniqueName: \"kubernetes.io/projected/b85be5a1-c949-4a43-9c96-a236c69e1b6c-kube-api-access-rdmv8\") pod \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " Nov 02 06:53:21.451595 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.451512 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-maas-api-tls\") pod \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " Nov 02 06:53:21.451685 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.451663 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85be5a1-c949-4a43-9c96-a236c69e1b6c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85be5a1-c949-4a43-9c96-a236c69e1b6c-openshift-service-ca\") pod \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\" (UID: \"b85be5a1-c949-4a43-9c96-a236c69e1b6c\") " Nov 02 06:53:21.452049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.452017 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85be5a1-c949-4a43-9c96-a236c69e1b6c-openshift-service-ca" pod "b85be5a1-c949-4a43-9c96-a236c69e1b6c" (UID: "b85be5a1-c949-4a43-9c96-a236c69e1b6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:21.453710 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.453683 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-maas-api-tls" pod "b85be5a1-c949-4a43-9c96-a236c69e1b6c" (UID: "b85be5a1-c949-4a43-9c96-a236c69e1b6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:21.453815 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.453712 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85be5a1-c949-4a43-9c96-a236c69e1b6c-kube-api-access-rdmv8" pod "b85be5a1-c949-4a43-9c96-a236c69e1b6c" (UID: "b85be5a1-c949-4a43-9c96-a236c69e1b6c"). InnerVolumeSpecName "kube-api-access-rdmv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:21.453815 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.453732 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-metrics-certs" pod "b85be5a1-c949-4a43-9c96-a236c69e1b6c" (UID: "b85be5a1-c949-4a43-9c96-a236c69e1b6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:21.553197 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.553161 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b85be5a1-c949-4a43-9c96-a236c69e1b6c-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:21.553197 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.553191 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:21.553197 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.553202 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdmv8\" (UniqueName: \"kubernetes.io/projected/b85be5a1-c949-4a43-9c96-a236c69e1b6c-kube-api-access-rdmv8\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:21.553433 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.553212 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b85be5a1-c949-4a43-9c96-a236c69e1b6c-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:21.762858 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.762824 5025 generic.go:356] "Generic (PLEG): container finished" podID="b85be5a1-c949-4a43-9c96-a236c69e1b6c" containerID="9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e" exitCode=0 Nov 02 06:53:21.763033 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.762867 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" Nov 02 06:53:21.763033 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.762907 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" event={"ID":"b85be5a1-c949-4a43-9c96-a236c69e1b6c","Type":"ContainerDied","Data":"9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e"} Nov 02 06:53:21.763033 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.762957 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" event={"ID":"b85be5a1-c949-4a43-9c96-a236c69e1b6c","Type":"ContainerDied","Data":"cbeeb8cddb89f5f64edb140002de4a30c2e178db1467a9018a7dce31ceb0a531"} Nov 02 06:53:21.763033 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.762979 5025 scope.go:122] "RemoveContainer" containerID="9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e" Nov 02 06:53:21.772850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.772829 5025 scope.go:122] "RemoveContainer" containerID="9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e" Nov 02 06:53:21.773130 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:53:21.773112 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e\": container with ID starting with 9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e not found: ID does not exist" containerID="9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e" Nov 02 06:53:21.773192 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.773141 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e"} err="failed to get container status \"9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e\": rpc error: code = NotFound desc = could not find container \"9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e\": container with ID starting with 9b2e07fd943454892db951d403597d97985b10ee35cdfd02c05cb1e15910945e not found: ID does not exist" Nov 02 06:53:21.919896 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.919854 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8"] Nov 02 06:53:21.926774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:21.926750 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8"] Nov 02 06:53:22.891915 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:22.891883 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:23.071087 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:23.071042 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:23.269271 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:23.269237 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:23.673932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:23.673896 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b85be5a1-c949-4a43-9c96-a236c69e1b6c" path="/var/lib/kubelet/pods/b85be5a1-c949-4a43-9c96-a236c69e1b6c/volumes" Nov 02 06:53:23.701346 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:23.701314 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:24.283531 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:24.283479 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e233a7-756f895c6c-m29s8" podUID="b85be5a1-c949-4a43-9c96-a236c69e1b6c" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.123:8443/health\": context deadline exceeded" Nov 02 06:53:25.889570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:25.889535 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:28.078883 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:28.078816 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:30.077514 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:30.077482 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:30.259796 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:30.259756 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:30.340614 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:30.340524 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx"] Nov 02 06:53:30.340810 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:30.340788 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" containerID="cri-o://7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f" gracePeriod=30 Nov 02 06:53:30.458754 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:30.457971 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:30.980144 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:30.980113 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:32.707594 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:53:32.707557 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:53:32.932106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:32.931969 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:34.178465 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:34.178428 5025 patch_prober.go:29] interesting pod/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:53:34.178911 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:34.178491 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:53:34.578856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:34.578774 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:34.759215 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:34.759180 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:35.598462 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.598437 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:53:35.664478 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.664452 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-certs\") pod \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " Nov 02 06:53:35.664649 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.664536 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-data\") pod \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " Nov 02 06:53:35.664649 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.664601 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-kube-api-access-kjtrl\" (UniqueName: \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-kube-api-access-kjtrl\") pod \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " Nov 02 06:53:35.664773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.664646 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-envoy\") pod \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " Nov 02 06:53:35.664773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.664676 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-podinfo\") pod \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " Nov 02 06:53:35.664773 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.664716 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-certs" pod "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" (UID: "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:35.664936 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.664775 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istiod-ca-cert\") pod \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " Nov 02 06:53:35.664936 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.664797 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-data" pod "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" (UID: "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:35.664936 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.664862 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-credential-socket\") pod \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " Nov 02 06:53:35.665119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.664961 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-socket\") pod \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " Nov 02 06:53:35.665119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.665045 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-token\" (UniqueName: \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-token\") pod \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\" (UID: \"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2\") " Nov 02 06:53:35.665226 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.665204 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-credential-socket" pod "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" (UID: "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:35.665281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.665228 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istiod-ca-cert" pod "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" (UID: "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:35.665731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.665434 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-socket" pod "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" (UID: "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:35.665908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.665773 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:35.665908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.665792 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:35.665908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.665807 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:35.665908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.665820 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:35.665908 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.665832 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:35.667127 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.667102 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-podinfo" pod "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" (UID: "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:53:35.667363 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.667347 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-kube-api-access-kjtrl" pod "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" (UID: "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2"). InnerVolumeSpecName "kube-api-access-kjtrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:35.667444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.667423 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-envoy" pod "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" (UID: "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:35.667482 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.667461 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-token" pod "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" (UID: "d4e66387-4e6c-4d71-9fd5-dd76e420dfd2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:35.766851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.766812 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:35.766851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.766850 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjtrl\" (UniqueName: \"kubernetes.io/projected/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-kube-api-access-kjtrl\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:35.767039 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.766867 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:35.767039 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.766881 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:35.834547 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.834512 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w"] Nov 02 06:53:35.845234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.845182 5025 generic.go:356] "Generic (PLEG): container finished" podID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerID="7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f" exitCode=0 Nov 02 06:53:35.845234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.845222 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" Nov 02 06:53:35.845455 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.845260 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" event={"ID":"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2","Type":"ContainerDied","Data":"7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f"} Nov 02 06:53:35.845455 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.845314 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx" event={"ID":"d4e66387-4e6c-4d71-9fd5-dd76e420dfd2","Type":"ContainerDied","Data":"49d65cc3382468aa0f04e2ddc097efcec4ee916b8a5bb7b39dc9d99623ae0a99"} Nov 02 06:53:35.845455 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.845341 5025 scope.go:122] "RemoveContainer" containerID="7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f" Nov 02 06:53:35.855313 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.855281 5025 scope.go:122] "RemoveContainer" containerID="58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3" Nov 02 06:53:35.863525 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.863504 5025 scope.go:122] "RemoveContainer" containerID="7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f" Nov 02 06:53:35.863780 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:53:35.863763 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f\": container with ID starting with 7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f not found: ID does not exist" containerID="7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f" Nov 02 06:53:35.863833 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.863794 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f"} err="failed to get container status \"7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f\": rpc error: code = NotFound desc = could not find container \"7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f\": container with ID starting with 7d65d9886860cf17ec5da11dda458f149e0a530c8989e05d8d3728c20014601f not found: ID does not exist" Nov 02 06:53:35.863833 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.863811 5025 scope.go:122] "RemoveContainer" containerID="58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3" Nov 02 06:53:35.864032 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:53:35.864017 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3\": container with ID starting with 58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3 not found: ID does not exist" containerID="58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3" Nov 02 06:53:35.864093 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:35.864037 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3"} err="failed to get container status \"58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3\": rpc error: code = NotFound desc = could not find container \"58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3\": container with ID starting with 58b5f128176b0f1f432d05a98755919aef5d01f57b485f0f3e8f13d9fdf30ba3 not found: ID does not exist" Nov 02 06:53:36.032037 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:36.031954 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:36.058566 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:36.058517 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx"] Nov 02 06:53:36.063706 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:36.063677 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e233a7-openshift-default-58b5979db5-6jgqx"] Nov 02 06:53:36.536029 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:36.535999 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:37.669214 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:37.669182 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" path="/var/lib/kubelet/pods/d4e66387-4e6c-4d71-9fd5-dd76e420dfd2/volumes" Nov 02 06:53:37.999551 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:37.999467 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:39.322104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.321044 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp"] Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.335832 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.335864 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.335880 5025 cpu_manager.go:414] "containerMap: removing container" podUID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" containerName="istio-proxy" Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.335888 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" containerName="istio-proxy" Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.335912 5025 cpu_manager.go:414] "containerMap: removing container" podUID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" containerName="istio-proxy" Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.335920 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" containerName="istio-proxy" Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.335935 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b85be5a1-c949-4a43-9c96-a236c69e1b6c" containerName="maas-api" Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.335942 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85be5a1-c949-4a43-9c96-a236c69e1b6c" containerName="maas-api" Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.335965 5025 cpu_manager.go:414] "containerMap: removing container" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.335973 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.335999 5025 cpu_manager.go:414] "containerMap: removing container" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" Nov 02 06:53:39.336103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.336008 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" Nov 02 06:53:39.336873 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.336160 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a4894f3-c78f-4acb-81b3-09dc3abcb0f7" containerName="istio-proxy" Nov 02 06:53:39.336873 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.336175 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" Nov 02 06:53:39.336873 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.336194 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" Nov 02 06:53:39.336873 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.336205 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="896cc162-c5d2-45c0-a86a-026df2b1b6f9" containerName="istio-proxy" Nov 02 06:53:39.336873 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.336215 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc2f04e7-114a-4ab0-8e5f-1035460391fd" containerName="istio-proxy" Nov 02 06:53:39.336873 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.336225 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="2637f05a-9e43-47c5-9e2c-8bffb1e58e4b" containerName="istio-proxy" Nov 02 06:53:39.336873 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.336234 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85be5a1-c949-4a43-9c96-a236c69e1b6c" containerName="maas-api" Nov 02 06:53:39.347111 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.341665 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.402105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.399789 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp"] Nov 02 06:53:39.654255 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.587712 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.663268 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.663227 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.663498 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.663475 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.796229 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.796197 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.796492 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.796466 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.806174 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.806146 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.818096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.806364 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.818329 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.818305 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.818526 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.818503 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwbtn\" (UniqueName: \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-kube-api-access-qwbtn\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.942099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.941765 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.942099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.941820 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.942099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.941886 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.942099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.941940 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.942099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.941974 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.942099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.942030 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.947101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.946109 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.947101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.946208 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwbtn\" (UniqueName: \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-kube-api-access-qwbtn\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.947101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.946316 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.947457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.947431 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-data\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.952099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.948065 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istiod-ca-cert\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.952099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.948348 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-credential-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.958099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.957545 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-certs\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.958309 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.958286 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-socket\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.958612 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.958591 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-envoy\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:39.971692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:39.971667 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-podinfo\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:40.021098 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:40.012881 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-token\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:40.031777 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:40.031742 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwbtn\" (UniqueName: \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-kube-api-access-qwbtn\") pod \"e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:40.108106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:40.105011 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:41.224814 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:41.224785 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp"] Nov 02 06:53:41.230972 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:41.230938 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:41.231031 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:41.231006 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:41.231066 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:41.231036 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:42.006202 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:42.006149 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" event={"ID":"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51","Type":"ContainerStarted","Data":"3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672"} Nov 02 06:53:42.006459 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:42.006435 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" event={"ID":"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51","Type":"ContainerStarted","Data":"88223c0a1a613cdf71940b39888a75b772396eda0d24900619be5208ac83a813"} Nov 02 06:53:42.113476 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:42.113445 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:42.113801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:42.113780 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:53:42.115395 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:42.115366 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 06:53:42.234799 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:42.234739 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" podStartSLOduration=3.234718701 podStartE2EDuration="3.234718701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:53:42.172970826 +0000 UTC m=+2198.621347602" watchObservedRunningTime="2026-11-02 06:53:42.234718701 +0000 UTC m=+2198.683095457" Nov 02 06:53:42.837196 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:42.837167 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:43.121782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:43.120608 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:43.121782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:43.121296 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:53:43.121782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:43.121344 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 06:53:43.945254 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:43.945218 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:44.108541 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:44.108505 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:53:44.108843 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:44.108814 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 06:53:44.140641 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:44.140611 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:45.123567 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:45.123533 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:53:45.124146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:45.124117 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 06:53:46.109894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:46.109455 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:53:46.109894 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:46.109526 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 06:53:47.108570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.108529 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:53:47.109129 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.108957 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 06:53:47.113210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.113186 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz"] Nov 02 06:53:47.113400 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.113382 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" podUID="8a402842-33d1-4046-9c1c-bc819bf91032" containerName="maas-api" containerID="cri-o://b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625" gracePeriod=30 Nov 02 06:53:47.270615 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.270569 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:47.472770 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.472745 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:53:47.569397 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.569360 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-maas-api-tls\") pod \"8a402842-33d1-4046-9c1c-bc819bf91032\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " Nov 02 06:53:47.569753 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.569727 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a402842-33d1-4046-9c1c-bc819bf91032-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a402842-33d1-4046-9c1c-bc819bf91032-openshift-service-ca\") pod \"8a402842-33d1-4046-9c1c-bc819bf91032\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " Nov 02 06:53:47.569962 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.569935 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a402842-33d1-4046-9c1c-bc819bf91032-kube-api-access-65l7q\" (UniqueName: \"kubernetes.io/projected/8a402842-33d1-4046-9c1c-bc819bf91032-kube-api-access-65l7q\") pod \"8a402842-33d1-4046-9c1c-bc819bf91032\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " Nov 02 06:53:47.570114 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.570100 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-metrics-certs\") pod \"8a402842-33d1-4046-9c1c-bc819bf91032\" (UID: \"8a402842-33d1-4046-9c1c-bc819bf91032\") " Nov 02 06:53:47.571261 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.571226 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a402842-33d1-4046-9c1c-bc819bf91032-openshift-service-ca" pod "8a402842-33d1-4046-9c1c-bc819bf91032" (UID: "8a402842-33d1-4046-9c1c-bc819bf91032"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:53:47.578497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.578467 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-metrics-certs" pod "8a402842-33d1-4046-9c1c-bc819bf91032" (UID: "8a402842-33d1-4046-9c1c-bc819bf91032"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:47.581244 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.581217 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-maas-api-tls" pod "8a402842-33d1-4046-9c1c-bc819bf91032" (UID: "8a402842-33d1-4046-9c1c-bc819bf91032"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:47.583552 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.583523 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a402842-33d1-4046-9c1c-bc819bf91032-kube-api-access-65l7q" pod "8a402842-33d1-4046-9c1c-bc819bf91032" (UID: "8a402842-33d1-4046-9c1c-bc819bf91032"). InnerVolumeSpecName "kube-api-access-65l7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:47.679694 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.672775 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:47.679694 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.672808 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a402842-33d1-4046-9c1c-bc819bf91032-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:47.679694 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.672823 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a402842-33d1-4046-9c1c-bc819bf91032-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:47.679694 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:47.672837 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65l7q\" (UniqueName: \"kubernetes.io/projected/8a402842-33d1-4046-9c1c-bc819bf91032-kube-api-access-65l7q\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:48.100119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.098782 5025 generic.go:356] "Generic (PLEG): container finished" podID="8a402842-33d1-4046-9c1c-bc819bf91032" containerID="b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625" exitCode=0 Nov 02 06:53:48.100119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.098931 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" event={"ID":"8a402842-33d1-4046-9c1c-bc819bf91032","Type":"ContainerDied","Data":"b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625"} Nov 02 06:53:48.100119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.098980 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" event={"ID":"8a402842-33d1-4046-9c1c-bc819bf91032","Type":"ContainerDied","Data":"73be65d43f06f2eeb6366fe749f720903b82c8232b93742dd9a1cc9a5ad345d4"} Nov 02 06:53:48.100119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.099007 5025 scope.go:122] "RemoveContainer" containerID="b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625" Nov 02 06:53:48.100119 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.099177 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz" Nov 02 06:53:48.107325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.106710 5025 patch_prober.go:29] interesting pod/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" start-of-body= Nov 02 06:53:48.107325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.106777 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.135:15021/healthz/ready\": dial tcp 10.133.0.135:15021: connect: connection refused" Nov 02 06:53:48.125156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.124622 5025 scope.go:122] "RemoveContainer" containerID="b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625" Nov 02 06:53:48.125156 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:53:48.125036 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625\": container with ID starting with b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625 not found: ID does not exist" containerID="b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625" Nov 02 06:53:48.125156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.125091 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625"} err="failed to get container status \"b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625\": rpc error: code = NotFound desc = could not find container \"b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625\": container with ID starting with b7898d3cf79bb1bf679270ced0b2472285486d558e65f7325661a1298ed4a625 not found: ID does not exist" Nov 02 06:53:48.279446 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.279408 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz"] Nov 02 06:53:48.320590 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.320551 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-bdcd80-7ff96cfcbd-cfqcz"] Nov 02 06:53:48.841969 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:48.841939 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:49.028617 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.027318 5025 patch_prober.go:29] interesting pod/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": read tcp 10.133.0.2:44994->10.133.0.133:15021: read: connection reset by peer" start-of-body= Nov 02 06:53:49.028617 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.027376 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": read tcp 10.133.0.2:44994->10.133.0.133:15021: read: connection reset by peer" Nov 02 06:53:49.106065 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.105984 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:49.121113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.117517 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:49.121113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.118712 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:49.121113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.119141 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:53:49.123386 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.123363 5025 generic.go:356] "Generic (PLEG): container finished" podID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerID="18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4" exitCode=0 Nov 02 06:53:49.123502 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.123439 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" event={"ID":"d0c0d533-68c3-4b4c-9769-fa8a193cba17","Type":"ContainerDied","Data":"18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4"} Nov 02 06:53:49.123915 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.123896 5025 scope.go:122] "RemoveContainer" containerID="18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4" Nov 02 06:53:49.125740 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.125710 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:49.126175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.125779 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:49.126175 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.125812 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:53:49.663142 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.663110 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:49.668897 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:49.668872 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a402842-33d1-4046-9c1c-bc819bf91032" path="/var/lib/kubelet/pods/8a402842-33d1-4046-9c1c-bc819bf91032/volumes" Nov 02 06:53:50.135211 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:50.135124 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" event={"ID":"d0c0d533-68c3-4b4c-9769-fa8a193cba17","Type":"ContainerStarted","Data":"2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301"} Nov 02 06:53:50.350427 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:50.350388 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:53:50.350635 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:50.350612 5025 patch_prober.go:29] interesting pod/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:53:50.350748 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:50.350726 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 06:53:50.857005 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:50.856969 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" containerID="cri-o://eae6ccf54650e0e3071749b7aa98e01f29b8230365b348476ee3131f6916fcc4" gracePeriod=45 Nov 02 06:53:50.858604 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:50.858580 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:51.008551 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.008504 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused" Nov 02 06:53:51.048491 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.048460 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:51.148005 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.147913 5025 generic.go:356] "Generic (PLEG): container finished" podID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerID="eae6ccf54650e0e3071749b7aa98e01f29b8230365b348476ee3131f6916fcc4" exitCode=0 Nov 02 06:53:51.149094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.149050 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" event={"ID":"e0264a41-8731-418b-ada7-bf8f950f2e22","Type":"ContainerDied","Data":"eae6ccf54650e0e3071749b7aa98e01f29b8230365b348476ee3131f6916fcc4"} Nov 02 06:53:51.182606 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.182587 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:53:51.227788 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.227758 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-dshm\") pod \"e0264a41-8731-418b-ada7-bf8f950f2e22\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " Nov 02 06:53:51.227945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.227892 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-home\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-home\") pod \"e0264a41-8731-418b-ada7-bf8f950f2e22\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " Nov 02 06:53:51.227945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.227939 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-model-cache\") pod \"e0264a41-8731-418b-ada7-bf8f950f2e22\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " Nov 02 06:53:51.228058 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.228036 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-tmp-dir\") pod \"e0264a41-8731-418b-ada7-bf8f950f2e22\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " Nov 02 06:53:51.228141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.228108 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0264a41-8731-418b-ada7-bf8f950f2e22-kube-api-access-76zgv\" (UniqueName: \"kubernetes.io/projected/e0264a41-8731-418b-ada7-bf8f950f2e22-kube-api-access-76zgv\") pod \"e0264a41-8731-418b-ada7-bf8f950f2e22\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " Nov 02 06:53:51.228204 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.228157 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0264a41-8731-418b-ada7-bf8f950f2e22-tls-certs\" (UniqueName: \"kubernetes.io/secret/e0264a41-8731-418b-ada7-bf8f950f2e22-tls-certs\") pod \"e0264a41-8731-418b-ada7-bf8f950f2e22\" (UID: \"e0264a41-8731-418b-ada7-bf8f950f2e22\") " Nov 02 06:53:51.228204 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.228171 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-home" pod "e0264a41-8731-418b-ada7-bf8f950f2e22" (UID: "e0264a41-8731-418b-ada7-bf8f950f2e22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.228307 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.228266 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-model-cache" pod "e0264a41-8731-418b-ada7-bf8f950f2e22" (UID: "e0264a41-8731-418b-ada7-bf8f950f2e22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.228307 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.228286 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-tmp-dir" pod "e0264a41-8731-418b-ada7-bf8f950f2e22" (UID: "e0264a41-8731-418b-ada7-bf8f950f2e22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.229023 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.229003 5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-home\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.229023 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.229022 5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-model-cache\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.229256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.229033 5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-tmp-dir\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.230309 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.230283 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-dshm" pod "e0264a41-8731-418b-ada7-bf8f950f2e22" (UID: "e0264a41-8731-418b-ada7-bf8f950f2e22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:53:51.230522 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.230496 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0264a41-8731-418b-ada7-bf8f950f2e22-tls-certs" pod "e0264a41-8731-418b-ada7-bf8f950f2e22" (UID: "e0264a41-8731-418b-ada7-bf8f950f2e22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:53:51.230608 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.230542 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0264a41-8731-418b-ada7-bf8f950f2e22-kube-api-access-76zgv" pod "e0264a41-8731-418b-ada7-bf8f950f2e22" (UID: "e0264a41-8731-418b-ada7-bf8f950f2e22"). InnerVolumeSpecName "kube-api-access-76zgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:53:51.330380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.330334 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76zgv\" (UniqueName: \"kubernetes.io/projected/e0264a41-8731-418b-ada7-bf8f950f2e22-kube-api-access-76zgv\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.330380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.330379 5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e0264a41-8731-418b-ada7-bf8f950f2e22-tls-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.330637 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.330394 5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e0264a41-8731-418b-ada7-bf8f950f2e22-dshm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:53:51.351050 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.351012 5025 patch_prober.go:29] interesting pod/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:53:51.351223 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:51.351121 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 06:53:52.155733 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:52.155655 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" Nov 02 06:53:52.155733 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:52.155695 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w" event={"ID":"e0264a41-8731-418b-ada7-bf8f950f2e22","Type":"ContainerDied","Data":"31e8686d30fcf252a1371953565da01b4a7847fd4c71a1482934761cec5f696d"} Nov 02 06:53:52.156258 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:52.155747 5025 scope.go:122] "RemoveContainer" containerID="eae6ccf54650e0e3071749b7aa98e01f29b8230365b348476ee3131f6916fcc4" Nov 02 06:53:52.315256 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:52.315223 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w"] Nov 02 06:53:52.350166 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:52.350139 5025 patch_prober.go:29] interesting pod/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:53:52.350318 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:52.350186 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 06:53:52.362239 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:52.362216 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fb1398/premium-w3-fb1398-kserve-64f46b7c96-m9q9w"] Nov 02 06:53:53.350941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:53.350910 5025 patch_prober.go:29] interesting pod/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:53:53.351356 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:53.350959 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 06:53:53.669599 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:53.669567 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" path="/var/lib/kubelet/pods/e0264a41-8731-418b-ada7-bf8f950f2e22/volumes" Nov 02 06:53:53.975153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:53.971602 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:54.350776 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:54.350691 5025 patch_prober.go:29] interesting pod/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:53:54.350776 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:54.350742 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 06:53:55.350813 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:55.350776 5025 patch_prober.go:29] interesting pod/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Nov 02 06:53:55.351274 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:55.350831 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Nov 02 06:53:55.555019 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:55.554983 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:55.739995 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:55.739959 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:55.820994 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:55.820963 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9"] Nov 02 06:53:55.821217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:55.821177 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerName="istio-proxy" containerID="cri-o://c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4" gracePeriod=30 Nov 02 06:53:55.927828 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:55.927604 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:53:56.354612 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:56.354581 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:53:56.354991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:56.354945 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:53:56.355576 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:56.355559 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 06:53:58.882070 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:53:58.882036 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:01.082877 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.082852 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:54:01.164986 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.164962 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-certs\") pod \"ad1a864e-1183-4c56-a21a-c76b3130d948\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " Nov 02 06:54:01.165153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165048 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-envoy\") pod \"ad1a864e-1183-4c56-a21a-c76b3130d948\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " Nov 02 06:54:01.165153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165101 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-data\") pod \"ad1a864e-1183-4c56-a21a-c76b3130d948\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " Nov 02 06:54:01.165153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165133 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-token\" (UniqueName: \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-token\") pod \"ad1a864e-1183-4c56-a21a-c76b3130d948\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " Nov 02 06:54:01.165284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165187 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-kube-api-access-pvvhc\" (UniqueName: \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-kube-api-access-pvvhc\") pod \"ad1a864e-1183-4c56-a21a-c76b3130d948\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " Nov 02 06:54:01.165355 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165327 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-certs" pod "ad1a864e-1183-4c56-a21a-c76b3130d948" (UID: "ad1a864e-1183-4c56-a21a-c76b3130d948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:54:01.165420 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165336 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad1a864e-1183-4c56-a21a-c76b3130d948-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad1a864e-1183-4c56-a21a-c76b3130d948-istiod-ca-cert\") pod \"ad1a864e-1183-4c56-a21a-c76b3130d948\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " Nov 02 06:54:01.165476 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165449 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-socket\") pod \"ad1a864e-1183-4c56-a21a-c76b3130d948\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " Nov 02 06:54:01.165476 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165458 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-data" pod "ad1a864e-1183-4c56-a21a-c76b3130d948" (UID: "ad1a864e-1183-4c56-a21a-c76b3130d948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:54:01.165668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165645 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-credential-socket\") pod \"ad1a864e-1183-4c56-a21a-c76b3130d948\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " Nov 02 06:54:01.165668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165659 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-socket" pod "ad1a864e-1183-4c56-a21a-c76b3130d948" (UID: "ad1a864e-1183-4c56-a21a-c76b3130d948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:54:01.165823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165675 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad1a864e-1183-4c56-a21a-c76b3130d948-istiod-ca-cert" pod "ad1a864e-1183-4c56-a21a-c76b3130d948" (UID: "ad1a864e-1183-4c56-a21a-c76b3130d948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:54:01.165823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165692 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-podinfo\") pod \"ad1a864e-1183-4c56-a21a-c76b3130d948\" (UID: \"ad1a864e-1183-4c56-a21a-c76b3130d948\") " Nov 02 06:54:01.165951 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.165931 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-credential-socket" pod "ad1a864e-1183-4c56-a21a-c76b3130d948" (UID: "ad1a864e-1183-4c56-a21a-c76b3130d948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:54:01.166305 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.166284 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad1a864e-1183-4c56-a21a-c76b3130d948-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.166408 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.166307 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.166408 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.166320 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.166408 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.166334 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.166408 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.166342 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.167446 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.167421 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-token" pod "ad1a864e-1183-4c56-a21a-c76b3130d948" (UID: "ad1a864e-1183-4c56-a21a-c76b3130d948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:54:01.167701 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.167679 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-envoy" pod "ad1a864e-1183-4c56-a21a-c76b3130d948" (UID: "ad1a864e-1183-4c56-a21a-c76b3130d948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:54:01.167782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.167755 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-podinfo" pod "ad1a864e-1183-4c56-a21a-c76b3130d948" (UID: "ad1a864e-1183-4c56-a21a-c76b3130d948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:54:01.167945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.167922 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-kube-api-access-pvvhc" pod "ad1a864e-1183-4c56-a21a-c76b3130d948" (UID: "ad1a864e-1183-4c56-a21a-c76b3130d948"). InnerVolumeSpecName "kube-api-access-pvvhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:54:01.211541 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.211514 5025 generic.go:356] "Generic (PLEG): container finished" podID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerID="c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4" exitCode=0 Nov 02 06:54:01.211659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.211561 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" Nov 02 06:54:01.211659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.211595 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" event={"ID":"ad1a864e-1183-4c56-a21a-c76b3130d948","Type":"ContainerDied","Data":"c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4"} Nov 02 06:54:01.211659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.211648 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9" event={"ID":"ad1a864e-1183-4c56-a21a-c76b3130d948","Type":"ContainerDied","Data":"2aac1f175b618049bc54b3f1be88110a48a5b80bf0a741c77e8e65ccb84bb24f"} Nov 02 06:54:01.211758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.211671 5025 scope.go:122] "RemoveContainer" containerID="c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4" Nov 02 06:54:01.221055 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.221038 5025 scope.go:122] "RemoveContainer" containerID="c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4" Nov 02 06:54:01.221330 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:54:01.221310 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4\": container with ID starting with c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4 not found: ID does not exist" containerID="c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4" Nov 02 06:54:01.221385 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.221339 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4"} err="failed to get container status \"c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4\": rpc error: code = NotFound desc = could not find container \"c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4\": container with ID starting with c02d71778d5c3b9f2d59fbff05ae9cfc2d50d0820b647ce9694c41261fa190f4 not found: ID does not exist" Nov 02 06:54:01.267801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.267768 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.267801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.267797 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.267801 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.267806 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.268016 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.267816 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvvhc\" (UniqueName: \"kubernetes.io/projected/ad1a864e-1183-4c56-a21a-c76b3130d948-kube-api-access-pvvhc\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:54:01.351238 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.351204 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9"] Nov 02 06:54:01.353744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.353719 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-bdcd80-openshift-default-6986d86f9f-hfcc9"] Nov 02 06:54:01.675781 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:01.675735 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" path="/var/lib/kubelet/pods/ad1a864e-1183-4c56-a21a-c76b3130d948/volumes" Nov 02 06:54:02.708295 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:54:02.708255 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:54:03.420384 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:03.420351 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:03.610097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:03.608766 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:07.250727 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:07.250696 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:08.358234 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:08.358196 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:08.541305 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:08.541272 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:12.142331 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:12.142289 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:12.322724 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:12.322690 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:13.093311 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:13.093273 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:13.688455 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:13.688420 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:17.700142 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:17.699775 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:18.837613 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:18.837585 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:19.026521 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:19.026486 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:20.182273 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:20.182226 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:22.367520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:22.367483 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:26.225311 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:26.225281 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:27.277140 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:27.268331 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:27.463604 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:27.463570 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:28.001653 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.001619 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:28.423296 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423257 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq"] Nov 02 06:54:28.423830 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423813 5025 cpu_manager.go:414] "containerMap: removing container" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" Nov 02 06:54:28.423874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423831 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" Nov 02 06:54:28.423874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423843 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" Nov 02 06:54:28.423874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423849 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e66387-4e6c-4d71-9fd5-dd76e420dfd2" containerName="istio-proxy" Nov 02 06:54:28.423874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423860 5025 cpu_manager.go:414] "containerMap: removing container" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerName="istio-proxy" Nov 02 06:54:28.423874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423866 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerName="istio-proxy" Nov 02 06:54:28.424025 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423879 5025 cpu_manager.go:414] "containerMap: removing container" podUID="8a402842-33d1-4046-9c1c-bc819bf91032" containerName="maas-api" Nov 02 06:54:28.424025 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423884 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a402842-33d1-4046-9c1c-bc819bf91032" containerName="maas-api" Nov 02 06:54:28.424025 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423971 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a402842-33d1-4046-9c1c-bc819bf91032" containerName="maas-api" Nov 02 06:54:28.424025 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423986 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0264a41-8731-418b-ada7-bf8f950f2e22" containerName="main" Nov 02 06:54:28.424025 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.423994 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad1a864e-1183-4c56-a21a-c76b3130d948" containerName="istio-proxy" Nov 02 06:54:28.427306 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.427291 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.478236 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.478205 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e6fde6-openshift-default-dockercfg-g9nd5\"" Nov 02 06:54:28.620708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.606631 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-socket\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.620708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.606692 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-credential-socket\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.620708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.606721 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-token\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.620708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.606756 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67b2cd6-2ab3-4f88-a576-691f8837773e-istiod-ca-cert\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.620708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.606848 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klfpb\" (UniqueName: \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-kube-api-access-klfpb\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.620708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.606938 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-podinfo\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.620708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.606967 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-data\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.620708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.607051 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-envoy\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.620708 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.607110 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-certs\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.622917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.622598 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:28.663333 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.663249 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq"] Nov 02 06:54:28.708748 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.708659 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-data\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.708748 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.708732 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-envoy\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.708941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.708761 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-certs\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.708941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.708866 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-socket\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.708941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.708898 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-credential-socket\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.708941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.708920 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-token\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.709090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.708966 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67b2cd6-2ab3-4f88-a576-691f8837773e-istiod-ca-cert\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.709090 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.709029 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klfpb\" (UniqueName: \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-kube-api-access-klfpb\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.709162 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.709087 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-data\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.709162 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.709102 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-podinfo\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.709629 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.709601 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67b2cd6-2ab3-4f88-a576-691f8837773e-istiod-ca-cert\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.710249 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.710225 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-certs\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.710392 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.710296 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-credential-socket\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.710392 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.710305 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-socket\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.711682 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.711662 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-podinfo\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.712001 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.711983 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-envoy\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.775242 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.775211 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klfpb\" (UniqueName: \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-kube-api-access-klfpb\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.778782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.778462 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-token\") pod \"e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:28.848430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:28.848398 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:29.040523 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:29.040425 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:29.127900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:29.127811 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:29.456499 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:29.456467 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq"] Nov 02 06:54:29.492461 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:54:29.492435 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf67b2cd6_2ab3_4f88_a576_691f8837773e.slice/crio-b2624ef5ec61b7a33a41efed6470e8b1e268c7ca9fc8293ea324be33d5ec5eed WatchSource:0}: Error finding container b2624ef5ec61b7a33a41efed6470e8b1e268c7ca9fc8293ea324be33d5ec5eed: Status 404 returned error can't find the container with id b2624ef5ec61b7a33a41efed6470e8b1e268c7ca9fc8293ea324be33d5ec5eed Nov 02 06:54:29.495103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:29.495041 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:54:29.495189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:29.495131 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:54:29.495189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:29.495160 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:54:30.374254 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:30.374220 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" event={"ID":"f67b2cd6-2ab3-4f88-a576-691f8837773e","Type":"ContainerStarted","Data":"19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae"} Nov 02 06:54:30.374439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:30.374272 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" event={"ID":"f67b2cd6-2ab3-4f88-a576-691f8837773e","Type":"ContainerStarted","Data":"b2624ef5ec61b7a33a41efed6470e8b1e268c7ca9fc8293ea324be33d5ec5eed"} Nov 02 06:54:30.535248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:30.535187 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" podStartSLOduration=2.5351734219999997 podStartE2EDuration="2.535173422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:54:30.532709795 +0000 UTC m=+2246.981086567" watchObservedRunningTime="2026-11-02 06:54:30.535173422 +0000 UTC m=+2246.983550177" Nov 02 06:54:31.040991 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:31.040961 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:31.046693 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:31.046668 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:31.380280 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:31.380203 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:31.381313 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:31.381295 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:54:31.905796 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:31.905758 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:32.706519 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:54:32.706476 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:54:33.252156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:33.252119 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:33.532327 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:33.532242 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:34.978614 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:34.978585 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:36.350818 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:36.350785 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:36.529406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:36.529358 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:37.036108 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:37.021988 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:38.516295 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:38.516262 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:41.208402 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:41.208364 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:41.397474 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:41.397434 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:41.700103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:41.697300 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:42.435830 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:42.435795 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:45.750269 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:45.750193 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:46.368537 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:46.368505 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:47.408751 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:47.408714 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:47.618488 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:47.618449 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:47.804505 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:47.804471 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:48.316971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:48.316937 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:49.659282 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:49.659238 5025 patch_prober.go:29] interesting pod/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:54:49.659782 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:49.659329 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:54:51.256258 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:51.256226 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp"] Nov 02 06:54:51.303249 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:51.303211 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4"] Nov 02 06:54:51.457235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:51.457204 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:51.587511 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:51.587476 5025 generic.go:356] "Generic (PLEG): container finished" podID="63732c47-cea0-4015-b099-8b7b465ccdff" containerID="7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2" exitCode=0 Nov 02 06:54:51.587662 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:51.587554 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" event={"ID":"63732c47-cea0-4015-b099-8b7b465ccdff","Type":"ContainerDied","Data":"7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2"} Nov 02 06:54:51.588066 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:51.588050 5025 scope.go:122] "RemoveContainer" containerID="7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2" Nov 02 06:54:51.589713 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:51.589686 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:54:51.589798 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:51.589752 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:54:51.589798 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:51.589783 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:54:52.690727 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:52.690690 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:52.790932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:52.790891 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" event={"ID":"63732c47-cea0-4015-b099-8b7b465ccdff","Type":"ContainerStarted","Data":"a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a"} Nov 02 06:54:53.139359 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:53.139276 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:54:53.139532 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:53.139461 5025 patch_prober.go:29] interesting pod/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:54:53.139616 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:53.139547 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 06:54:54.139850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:54.139813 5025 patch_prober.go:29] interesting pod/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:54:54.140264 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:54.139865 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 06:54:54.886763 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:54.886721 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:55.082206 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:55.082156 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:55.139862 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:55.139773 5025 patch_prober.go:29] interesting pod/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:54:55.139862 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:55.139855 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 06:54:55.167411 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:55.167383 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp"] Nov 02 06:54:55.167600 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:55.167580 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerName="istio-proxy" containerID="cri-o://3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672" gracePeriod=30 Nov 02 06:54:55.172928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:55.172903 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz"] Nov 02 06:54:55.173150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:55.173128 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" containerID="cri-o://1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe" gracePeriod=30 Nov 02 06:54:55.304679 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:55.304638 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:56.139848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:56.139816 5025 patch_prober.go:29] interesting pod/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:54:56.140016 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:56.139871 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 06:54:57.139411 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:57.139380 5025 patch_prober.go:29] interesting pod/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Nov 02 06:54:57.139583 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:57.139434 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Nov 02 06:54:57.378311 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:57.378264 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:58.143912 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:58.143882 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:54:58.144224 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:58.144202 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:54:58.144865 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:58.144847 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 06:54:58.594317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:58.594279 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:54:58.805337 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:54:58.805308 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:00.443677 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.443618 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:55:00.478370 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.478346 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:55:00.517895 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.517871 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-token\" (UniqueName: \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-token\") pod \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " Nov 02 06:55:00.518046 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.517902 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-socket\") pod \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " Nov 02 06:55:00.518046 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.517948 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-kube-api-access-7xgjw\" (UniqueName: \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-kube-api-access-7xgjw\") pod \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " Nov 02 06:55:00.518046 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518004 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-certs\") pod \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " Nov 02 06:55:00.518046 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518035 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istiod-ca-cert\") pod \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " Nov 02 06:55:00.518308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518112 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-envoy\") pod \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " Nov 02 06:55:00.518308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518179 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-data\") pod \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " Nov 02 06:55:00.518308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518209 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-socket" pod "6d9c4a77-9cc7-4478-966e-e0762e9c5c97" (UID: "6d9c4a77-9cc7-4478-966e-e0762e9c5c97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.518308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518213 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-certs\") pod \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " Nov 02 06:55:00.518308 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518284 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-credential-socket\") pod \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " Nov 02 06:55:00.518570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518313 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-kube-api-access-qwbtn\" (UniqueName: \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-kube-api-access-qwbtn\") pod \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " Nov 02 06:55:00.518570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518326 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-certs" pod "6d9c4a77-9cc7-4478-966e-e0762e9c5c97" (UID: "6d9c4a77-9cc7-4478-966e-e0762e9c5c97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.518570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518365 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-socket\") pod \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " Nov 02 06:55:00.518570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518435 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-credential-socket\") pod \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " Nov 02 06:55:00.518570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518489 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-envoy\") pod \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " Nov 02 06:55:00.518570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518523 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istiod-ca-cert\") pod \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " Nov 02 06:55:00.518570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518550 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-podinfo\") pod \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " Nov 02 06:55:00.518926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518435 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-certs" pod "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" (UID: "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.518926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518464 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istiod-ca-cert" pod "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" (UID: "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:55:00.518926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518573 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-credential-socket" pod "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" (UID: "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.518926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518695 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-data" pod "6d9c4a77-9cc7-4478-966e-e0762e9c5c97" (UID: "6d9c4a77-9cc7-4478-966e-e0762e9c5c97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.518926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518734 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-socket" pod "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" (UID: "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.518926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518605 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-podinfo\") pod \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " Nov 02 06:55:00.518926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518798 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-credential-socket" pod "6d9c4a77-9cc7-4478-966e-e0762e9c5c97" (UID: "6d9c4a77-9cc7-4478-966e-e0762e9c5c97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.518926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518842 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-token\" (UniqueName: \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-token\") pod \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\" (UID: \"6d9c4a77-9cc7-4478-966e-e0762e9c5c97\") " Nov 02 06:55:00.518926 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.518876 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-data\") pod \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\" (UID: \"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51\") " Nov 02 06:55:00.519502 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.519480 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.519564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.519507 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.519564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.519522 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.519564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.519537 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.519564 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.519550 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.519770 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.519568 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.519770 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.519582 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.519770 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.519594 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.520361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.519821 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-data" pod "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" (UID: "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.520361 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.519842 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istiod-ca-cert" pod "6d9c4a77-9cc7-4478-966e-e0762e9c5c97" (UID: "6d9c4a77-9cc7-4478-966e-e0762e9c5c97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:55:00.521189 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.520895 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-kube-api-access-qwbtn" pod "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" (UID: "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51"). InnerVolumeSpecName "kube-api-access-qwbtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:00.521282 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.521231 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-envoy" pod "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" (UID: "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.521331 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.521309 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-token" pod "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" (UID: "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:00.521424 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.521400 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-podinfo" pod "6d9c4a77-9cc7-4478-966e-e0762e9c5c97" (UID: "6d9c4a77-9cc7-4478-966e-e0762e9c5c97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:55:00.521570 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.521553 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-kube-api-access-7xgjw" pod "6d9c4a77-9cc7-4478-966e-e0762e9c5c97" (UID: "6d9c4a77-9cc7-4478-966e-e0762e9c5c97"). InnerVolumeSpecName "kube-api-access-7xgjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:00.521795 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.521772 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-envoy" pod "6d9c4a77-9cc7-4478-966e-e0762e9c5c97" (UID: "6d9c4a77-9cc7-4478-966e-e0762e9c5c97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:00.522015 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.521997 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-podinfo" pod "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" (UID: "ea346bc6-baf2-4fdc-ae07-bc4f8d010b51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:55:00.522890 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.522871 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-token" pod "6d9c4a77-9cc7-4478-966e-e0762e9c5c97" (UID: "6d9c4a77-9cc7-4478-966e-e0762e9c5c97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:00.620692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.620659 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwbtn\" (UniqueName: \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-kube-api-access-qwbtn\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.620692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.620686 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.620692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.620695 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.620692 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.620704 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.621040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.620716 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.621040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.620729 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.621040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.620740 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.621040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.620748 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.621040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.620758 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xgjw\" (UniqueName: \"kubernetes.io/projected/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-kube-api-access-7xgjw\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.621040 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.620766 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d9c4a77-9cc7-4478-966e-e0762e9c5c97-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:00.873940 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.873869 5025 generic.go:356] "Generic (PLEG): container finished" podID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerID="3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672" exitCode=0 Nov 02 06:55:00.873940 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.873929 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" Nov 02 06:55:00.874150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.873962 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" event={"ID":"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51","Type":"ContainerDied","Data":"3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672"} Nov 02 06:55:00.874150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.874014 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp" event={"ID":"ea346bc6-baf2-4fdc-ae07-bc4f8d010b51","Type":"ContainerDied","Data":"88223c0a1a613cdf71940b39888a75b772396eda0d24900619be5208ac83a813"} Nov 02 06:55:00.874150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.874047 5025 scope.go:122] "RemoveContainer" containerID="3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672" Nov 02 06:55:00.875939 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.875902 5025 generic.go:356] "Generic (PLEG): container finished" podID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerID="1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe" exitCode=0 Nov 02 06:55:00.876043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.875949 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" Nov 02 06:55:00.876043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.875976 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" event={"ID":"6d9c4a77-9cc7-4478-966e-e0762e9c5c97","Type":"ContainerDied","Data":"1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe"} Nov 02 06:55:00.876043 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.876031 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz" event={"ID":"6d9c4a77-9cc7-4478-966e-e0762e9c5c97","Type":"ContainerDied","Data":"633f313d498f630b7aff4a1d5cef07ff6ee0af8defb8aed441ea14ba130913d5"} Nov 02 06:55:00.883900 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.883879 5025 scope.go:122] "RemoveContainer" containerID="3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672" Nov 02 06:55:00.884159 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:55:00.884139 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672\": container with ID starting with 3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672 not found: ID does not exist" containerID="3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672" Nov 02 06:55:00.884253 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.884166 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672"} err="failed to get container status \"3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672\": rpc error: code = NotFound desc = could not find container \"3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672\": container with ID starting with 3b6f60aed695408b55211487f7e612320de8679b0aae9ddb30d63fcb02157672 not found: ID does not exist" Nov 02 06:55:00.884253 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.884185 5025 scope.go:122] "RemoveContainer" containerID="1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe" Nov 02 06:55:00.893676 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.893660 5025 scope.go:122] "RemoveContainer" containerID="be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8" Nov 02 06:55:00.901769 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.901754 5025 scope.go:122] "RemoveContainer" containerID="1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe" Nov 02 06:55:00.902039 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:55:00.902015 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe\": container with ID starting with 1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe not found: ID does not exist" containerID="1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe" Nov 02 06:55:00.902154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.902045 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe"} err="failed to get container status \"1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe\": rpc error: code = NotFound desc = could not find container \"1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe\": container with ID starting with 1dd53a10fd8cc7623b3c4fe24bb7ff6d6ee7ec8016976f6d71554c690f119abe not found: ID does not exist" Nov 02 06:55:00.902154 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.902063 5025 scope.go:122] "RemoveContainer" containerID="be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8" Nov 02 06:55:00.902359 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:55:00.902342 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8\": container with ID starting with be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8 not found: ID does not exist" containerID="be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8" Nov 02 06:55:00.902401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:00.902364 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8"} err="failed to get container status \"be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8\": rpc error: code = NotFound desc = could not find container \"be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8\": container with ID starting with be5656c1efb3accad1ec8da380d69a2ad5b2e904e79acda66b1b0a9a164934e8 not found: ID does not exist" Nov 02 06:55:01.033210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:01.033181 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp"] Nov 02 06:55:01.037978 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:01.037942 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-pjwnp"] Nov 02 06:55:01.142437 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:01.142364 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz"] Nov 02 06:55:01.148817 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:01.148792 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-dcf9c3-openshift-default-6df64cd48c-jd2hz"] Nov 02 06:55:01.338982 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:01.338953 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:01.670096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:01.670038 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" path="/var/lib/kubelet/pods/6d9c4a77-9cc7-4478-966e-e0762e9c5c97/volumes" Nov 02 06:55:01.670632 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:01.670616 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" path="/var/lib/kubelet/pods/ea346bc6-baf2-4fdc-ae07-bc4f8d010b51/volumes" Nov 02 06:55:02.707264 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:55:02.707226 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:55:02.840525 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:02.840494 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:03.062965 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:03.062933 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:03.243742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:03.243703 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:03.714391 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:03.714357 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:06.278150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.278112 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" containerID="cri-o://d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f" gracePeriod=45 Nov 02 06:55:06.323000 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.322966 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" containerID="cri-o://48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11" gracePeriod=45 Nov 02 06:55:06.485364 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.484983 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.45:8000/ready\": dial tcp 10.133.0.45:8000: connect: connection refused" Nov 02 06:55:06.579049 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.579019 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:55:06.604905 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.604880 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-home\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-home\") pod \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " Nov 02 06:55:06.605044 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.604934 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tls-certs\") pod \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " Nov 02 06:55:06.605044 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.604974 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac70ab7-07cd-4046-8882-fbf50037c5ba-kube-api-access-flz9h\" (UniqueName: \"kubernetes.io/projected/8ac70ab7-07cd-4046-8882-fbf50037c5ba-kube-api-access-flz9h\") pod \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " Nov 02 06:55:06.605044 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.605039 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tmp-dir\") pod \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " Nov 02 06:55:06.605246 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.605203 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-dshm\") pod \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " Nov 02 06:55:06.605302 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.605261 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-home" pod "8ac70ab7-07cd-4046-8882-fbf50037c5ba" (UID: "8ac70ab7-07cd-4046-8882-fbf50037c5ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.605302 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.605278 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-model-cache\") pod \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\" (UID: \"8ac70ab7-07cd-4046-8882-fbf50037c5ba\") " Nov 02 06:55:06.605593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.605569 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-model-cache" pod "8ac70ab7-07cd-4046-8882-fbf50037c5ba" (UID: "8ac70ab7-07cd-4046-8882-fbf50037c5ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.605668 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.605631 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tmp-dir" pod "8ac70ab7-07cd-4046-8882-fbf50037c5ba" (UID: "8ac70ab7-07cd-4046-8882-fbf50037c5ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.605878 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.605857 5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-home\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.605951 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.605881 5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tmp-dir\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.605951 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.605897 5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-model-cache\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.607497 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.607471 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-dshm" pod "8ac70ab7-07cd-4046-8882-fbf50037c5ba" (UID: "8ac70ab7-07cd-4046-8882-fbf50037c5ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.607583 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.607512 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac70ab7-07cd-4046-8882-fbf50037c5ba-kube-api-access-flz9h" pod "8ac70ab7-07cd-4046-8882-fbf50037c5ba" (UID: "8ac70ab7-07cd-4046-8882-fbf50037c5ba"). InnerVolumeSpecName "kube-api-access-flz9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:06.607583 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.607566 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tls-certs" pod "8ac70ab7-07cd-4046-8882-fbf50037c5ba" (UID: "8ac70ab7-07cd-4046-8882-fbf50037c5ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:55:06.613467 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.613449 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:55:06.706822 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.706789 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90419b03-68d9-4a02-845b-b600b2bb10f7-kube-api-access-24vhf\" (UniqueName: \"kubernetes.io/projected/90419b03-68d9-4a02-845b-b600b2bb10f7-kube-api-access-24vhf\") pod \"90419b03-68d9-4a02-845b-b600b2bb10f7\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " Nov 02 06:55:06.707004 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.706878 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-model-cache\") pod \"90419b03-68d9-4a02-845b-b600b2bb10f7\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " Nov 02 06:55:06.707104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.707011 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-home\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-home\") pod \"90419b03-68d9-4a02-845b-b600b2bb10f7\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " Nov 02 06:55:06.707104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.707054 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90419b03-68d9-4a02-845b-b600b2bb10f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/90419b03-68d9-4a02-845b-b600b2bb10f7-tls-certs\") pod \"90419b03-68d9-4a02-845b-b600b2bb10f7\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " Nov 02 06:55:06.707104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.707102 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-tmp-dir\") pod \"90419b03-68d9-4a02-845b-b600b2bb10f7\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " Nov 02 06:55:06.707285 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.707157 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-dshm\") pod \"90419b03-68d9-4a02-845b-b600b2bb10f7\" (UID: \"90419b03-68d9-4a02-845b-b600b2bb10f7\") " Nov 02 06:55:06.707285 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.707189 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-model-cache" pod "90419b03-68d9-4a02-845b-b600b2bb10f7" (UID: "90419b03-68d9-4a02-845b-b600b2bb10f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.708987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.707425 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-tmp-dir" pod "90419b03-68d9-4a02-845b-b600b2bb10f7" (UID: "90419b03-68d9-4a02-845b-b600b2bb10f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.708987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.707919 5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-tmp-dir\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.708987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.707940 5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8ac70ab7-07cd-4046-8882-fbf50037c5ba-dshm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.708987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.707953 5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-model-cache\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.708987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.707968 5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8ac70ab7-07cd-4046-8882-fbf50037c5ba-tls-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.708987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.707983 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flz9h\" (UniqueName: \"kubernetes.io/projected/8ac70ab7-07cd-4046-8882-fbf50037c5ba-kube-api-access-flz9h\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.709518 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.709493 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90419b03-68d9-4a02-845b-b600b2bb10f7-tls-certs" pod "90419b03-68d9-4a02-845b-b600b2bb10f7" (UID: "90419b03-68d9-4a02-845b-b600b2bb10f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:55:06.709630 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.709527 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-home" pod "90419b03-68d9-4a02-845b-b600b2bb10f7" (UID: "90419b03-68d9-4a02-845b-b600b2bb10f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.710039 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.710022 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90419b03-68d9-4a02-845b-b600b2bb10f7-kube-api-access-24vhf" pod "90419b03-68d9-4a02-845b-b600b2bb10f7" (UID: "90419b03-68d9-4a02-845b-b600b2bb10f7"). InnerVolumeSpecName "kube-api-access-24vhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:06.710323 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.710303 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-dshm" pod "90419b03-68d9-4a02-845b-b600b2bb10f7" (UID: "90419b03-68d9-4a02-845b-b600b2bb10f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:06.809352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.809270 5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-home\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.809352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.809300 5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90419b03-68d9-4a02-845b-b600b2bb10f7-tls-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.809352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.809309 5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90419b03-68d9-4a02-845b-b600b2bb10f7-dshm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.809352 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.809318 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24vhf\" (UniqueName: \"kubernetes.io/projected/90419b03-68d9-4a02-845b-b600b2bb10f7-kube-api-access-24vhf\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:06.918458 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.918420 5025 generic.go:356] "Generic (PLEG): container finished" podID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerID="d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f" exitCode=0 Nov 02 06:55:06.918625 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.918467 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" Nov 02 06:55:06.918625 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.918502 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" event={"ID":"8ac70ab7-07cd-4046-8882-fbf50037c5ba","Type":"ContainerDied","Data":"d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f"} Nov 02 06:55:06.918625 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.918542 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp" event={"ID":"8ac70ab7-07cd-4046-8882-fbf50037c5ba","Type":"ContainerDied","Data":"a1cfb101f45432daefed22caee2c2870404e0140d27a014fdfe12d8594a6a34d"} Nov 02 06:55:06.918625 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.918563 5025 scope.go:122] "RemoveContainer" containerID="d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f" Nov 02 06:55:06.919840 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.919823 5025 generic.go:356] "Generic (PLEG): container finished" podID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerID="48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11" exitCode=0 Nov 02 06:55:06.919945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.919882 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" Nov 02 06:55:06.919945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.919905 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" event={"ID":"90419b03-68d9-4a02-845b-b600b2bb10f7","Type":"ContainerDied","Data":"48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11"} Nov 02 06:55:06.919945 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.919934 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4" event={"ID":"90419b03-68d9-4a02-845b-b600b2bb10f7","Type":"ContainerDied","Data":"d2074fdf565f8cfb4720c201ce25445c7186e3d9b002abb7640f3ba24c3f666e"} Nov 02 06:55:06.928212 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.928192 5025 scope.go:122] "RemoveContainer" containerID="d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f" Nov 02 06:55:06.928546 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:55:06.928512 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f\": container with ID starting with d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f not found: ID does not exist" containerID="d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f" Nov 02 06:55:06.928672 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.928554 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f"} err="failed to get container status \"d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f\": rpc error: code = NotFound desc = could not find container \"d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f\": container with ID starting with d7bc14d71a1d06e526447813a119e3a6377d6369c1a21933f698c8b89736c78f not found: ID does not exist" Nov 02 06:55:06.928672 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.928580 5025 scope.go:122] "RemoveContainer" containerID="48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11" Nov 02 06:55:06.937146 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.937129 5025 scope.go:122] "RemoveContainer" containerID="48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11" Nov 02 06:55:06.937416 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:55:06.937395 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11\": container with ID starting with 48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11 not found: ID does not exist" containerID="48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11" Nov 02 06:55:06.937468 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:06.937423 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11"} err="failed to get container status \"48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11\": rpc error: code = NotFound desc = could not find container \"48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11\": container with ID starting with 48f7f8939a8ffc86baeae236d68352f53ddfb594811245a608aafff18e20da11 not found: ID does not exist" Nov 02 06:55:07.076979 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:07.076876 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4"] Nov 02 06:55:07.080103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:07.080061 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/premium-w4-680b7b-kserve-8668b4f54f-2jxw4"] Nov 02 06:55:07.216187 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:07.216046 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp"] Nov 02 06:55:07.219842 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:07.219749 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-680b7b/distinct-w4-680b7b-kserve-6958688496-lgpqp"] Nov 02 06:55:07.669130 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:07.669061 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" path="/var/lib/kubelet/pods/8ac70ab7-07cd-4046-8882-fbf50037c5ba/volumes" Nov 02 06:55:07.669560 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:07.669545 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" path="/var/lib/kubelet/pods/90419b03-68d9-4a02-845b-b600b2bb10f7/volumes" Nov 02 06:55:08.183062 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:08.182984 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:09.964848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:09.964815 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:10.163301 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:10.163270 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:10.258865 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:10.258786 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm"] Nov 02 06:55:10.259030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:10.258973 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" podUID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" containerName="istio-proxy" containerID="cri-o://9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385" gracePeriod=30 Nov 02 06:55:10.377101 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:10.363521 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:13.153870 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:13.153834 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:13.351279 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:13.351177 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:15.528377 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.528353 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:55:15.582282 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582211 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-token\") pod \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " Nov 02 06:55:15.582414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582307 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-socket\") pod \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " Nov 02 06:55:15.582414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582336 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-data\") pod \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " Nov 02 06:55:15.582414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582357 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-kube-api-access-v48hf\" (UniqueName: \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-kube-api-access-v48hf\") pod \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " Nov 02 06:55:15.582414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582376 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-podinfo\") pod \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " Nov 02 06:55:15.582591 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582451 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istiod-ca-cert\") pod \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " Nov 02 06:55:15.582591 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582524 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-credential-socket\") pod \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " Nov 02 06:55:15.582591 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582552 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-envoy\") pod \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " Nov 02 06:55:15.582742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582617 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-certs\") pod \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\" (UID: \"ed9a87c7-20ed-4343-9eb7-2b713d4cd569\") " Nov 02 06:55:15.582742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582623 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-socket" pod "ed9a87c7-20ed-4343-9eb7-2b713d4cd569" (UID: "ed9a87c7-20ed-4343-9eb7-2b713d4cd569"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:15.582742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582656 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-data" pod "ed9a87c7-20ed-4343-9eb7-2b713d4cd569" (UID: "ed9a87c7-20ed-4343-9eb7-2b713d4cd569"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:15.582901 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582842 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-certs" pod "ed9a87c7-20ed-4343-9eb7-2b713d4cd569" (UID: "ed9a87c7-20ed-4343-9eb7-2b713d4cd569"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:15.582901 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582884 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-credential-socket" pod "ed9a87c7-20ed-4343-9eb7-2b713d4cd569" (UID: "ed9a87c7-20ed-4343-9eb7-2b713d4cd569"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:15.583004 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.582954 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istiod-ca-cert" pod "ed9a87c7-20ed-4343-9eb7-2b713d4cd569" (UID: "ed9a87c7-20ed-4343-9eb7-2b713d4cd569"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:55:15.583342 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.583320 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.583414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.583349 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.583414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.583365 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.583414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.583378 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.583414 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.583392 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.584603 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.584564 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-podinfo" pod "ed9a87c7-20ed-4343-9eb7-2b713d4cd569" (UID: "ed9a87c7-20ed-4343-9eb7-2b713d4cd569"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:55:15.584688 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.584641 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-envoy" pod "ed9a87c7-20ed-4343-9eb7-2b713d4cd569" (UID: "ed9a87c7-20ed-4343-9eb7-2b713d4cd569"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:55:15.584786 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.584768 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-token" pod "ed9a87c7-20ed-4343-9eb7-2b713d4cd569" (UID: "ed9a87c7-20ed-4343-9eb7-2b713d4cd569"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:15.585089 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.584883 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-kube-api-access-v48hf" pod "ed9a87c7-20ed-4343-9eb7-2b713d4cd569" (UID: "ed9a87c7-20ed-4343-9eb7-2b713d4cd569"). InnerVolumeSpecName "kube-api-access-v48hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:55:15.685054 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.685025 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.685054 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.685053 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.685247 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.685063 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v48hf\" (UniqueName: \"kubernetes.io/projected/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-kube-api-access-v48hf\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:15.685247 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:15.685087 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed9a87c7-20ed-4343-9eb7-2b713d4cd569-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:55:16.014143 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:16.014108 5025 generic.go:356] "Generic (PLEG): container finished" podID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" containerID="9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385" exitCode=0 Nov 02 06:55:16.014294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:16.014170 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" event={"ID":"ed9a87c7-20ed-4343-9eb7-2b713d4cd569","Type":"ContainerDied","Data":"9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385"} Nov 02 06:55:16.014294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:16.014200 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" Nov 02 06:55:16.014294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:16.014207 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm" event={"ID":"ed9a87c7-20ed-4343-9eb7-2b713d4cd569","Type":"ContainerDied","Data":"8468071a0eb56adbd1a43fd1342b62948bc7cd938392ccef336121ad84da93ba"} Nov 02 06:55:16.014294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:16.014229 5025 scope.go:122] "RemoveContainer" containerID="9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385" Nov 02 06:55:16.023340 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:16.023307 5025 scope.go:122] "RemoveContainer" containerID="9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385" Nov 02 06:55:16.023615 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:55:16.023596 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385\": container with ID starting with 9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385 not found: ID does not exist" containerID="9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385" Nov 02 06:55:16.023673 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:16.023623 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385"} err="failed to get container status \"9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385\": rpc error: code = NotFound desc = could not find container \"9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385\": container with ID starting with 9d3531705ca61e6e7b27b9095a25a937f70f3d6f366364054d240416ec409385 not found: ID does not exist" Nov 02 06:55:16.169786 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:16.169746 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm"] Nov 02 06:55:16.173689 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:16.173648 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fb1398-openshift-default-74d679c8f7-j2jlm"] Nov 02 06:55:17.668924 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:17.668887 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" path="/var/lib/kubelet/pods/ed9a87c7-20ed-4343-9eb7-2b713d4cd569/volumes" Nov 02 06:55:17.987248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:17.986681 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:22.540677 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:22.540635 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:26.536707 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.536672 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:26.712712 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.712674 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn"] Nov 02 06:55:26.713218 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713205 5025 cpu_manager.go:414] "containerMap: removing container" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" Nov 02 06:55:26.713277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713219 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" Nov 02 06:55:26.713277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713233 5025 cpu_manager.go:414] "containerMap: removing container" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" Nov 02 06:55:26.713277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713238 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" Nov 02 06:55:26.713277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713261 5025 cpu_manager.go:414] "containerMap: removing container" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" Nov 02 06:55:26.713277 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713266 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" Nov 02 06:55:26.713444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713280 5025 cpu_manager.go:414] "containerMap: removing container" podUID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" containerName="istio-proxy" Nov 02 06:55:26.713444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713285 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" containerName="istio-proxy" Nov 02 06:55:26.713444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713293 5025 cpu_manager.go:414] "containerMap: removing container" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" Nov 02 06:55:26.713444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713298 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" Nov 02 06:55:26.713444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713305 5025 cpu_manager.go:414] "containerMap: removing container" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerName="istio-proxy" Nov 02 06:55:26.713444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713309 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerName="istio-proxy" Nov 02 06:55:26.713444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713375 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" Nov 02 06:55:26.713444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713384 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed9a87c7-20ed-4343-9eb7-2b713d4cd569" containerName="istio-proxy" Nov 02 06:55:26.713444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713392 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac70ab7-07cd-4046-8882-fbf50037c5ba" containerName="main" Nov 02 06:55:26.713444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713397 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea346bc6-baf2-4fdc-ae07-bc4f8d010b51" containerName="istio-proxy" Nov 02 06:55:26.713444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.713404 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="90419b03-68d9-4a02-845b-b600b2bb10f7" containerName="main" Nov 02 06:55:26.718262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.718244 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:26.771112 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.771087 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn"] Nov 02 06:55:26.815265 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.815189 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e6fde6-serving-cert\"" Nov 02 06:55:26.816776 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.816749 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-metrics-certs\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:26.816844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.816794 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-maas-api-tls\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:26.816844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.816814 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0881f6d1-1062-471e-9d54-be23d9688330-openshift-service-ca\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:26.816944 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.816921 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hccz8\" (UniqueName: \"kubernetes.io/projected/0881f6d1-1062-471e-9d54-be23d9688330-kube-api-access-hccz8\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:26.917871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.917835 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-metrics-certs\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:26.917871 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.917873 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-maas-api-tls\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:26.918145 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.917893 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0881f6d1-1062-471e-9d54-be23d9688330-openshift-service-ca\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:26.918145 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.917969 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hccz8\" (UniqueName: \"kubernetes.io/projected/0881f6d1-1062-471e-9d54-be23d9688330-kube-api-access-hccz8\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:26.918145 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:55:26.917977 5025 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-serving-cert: secret "maas-api-e2e-api-b-e6fde6-serving-cert" not found Nov 02 06:55:26.918145 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:55:26.918046 5025 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-maas-api-tls podName:0881f6d1-1062-471e-9d54-be23d9688330 nodeName:}" failed. No retries permitted until 2026-11-02 06:55:27.418030371 +0000 UTC m=+2303.866407117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-maas-api-tls") pod "maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" (UID: "0881f6d1-1062-471e-9d54-be23d9688330") : secret "maas-api-e2e-api-b-e6fde6-serving-cert" not found Nov 02 06:55:26.918516 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.918493 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0881f6d1-1062-471e-9d54-be23d9688330-openshift-service-ca\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:26.920375 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.920356 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-metrics-certs\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:26.955284 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.955253 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:26.972870 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:26.972839 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hccz8\" (UniqueName: \"kubernetes.io/projected/0881f6d1-1062-471e-9d54-be23d9688330-kube-api-access-hccz8\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:27.142244 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:27.134625 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:27.425387 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:27.425360 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-maas-api-tls\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:27.427729 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:27.427706 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-maas-api-tls\") pod \"maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:27.631072 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:27.631040 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:28.000316 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:28.000289 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn"] Nov 02 06:55:28.001643 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:55:28.001616 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0881f6d1_1062_471e_9d54_be23d9688330.slice/crio-7bf116ac438ed14eda2428b761c8af72e08a02b4e04df30774388952974d39b1 WatchSource:0}: Error finding container 7bf116ac438ed14eda2428b761c8af72e08a02b4e04df30774388952974d39b1: Status 404 returned error can't find the container with id 7bf116ac438ed14eda2428b761c8af72e08a02b4e04df30774388952974d39b1 Nov 02 06:55:28.002960 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:28.002942 5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 06:55:28.089153 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:28.089122 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" event={"ID":"0881f6d1-1062-471e-9d54-be23d9688330","Type":"ContainerStarted","Data":"7bf116ac438ed14eda2428b761c8af72e08a02b4e04df30774388952974d39b1"} Nov 02 06:55:28.947759 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:28.947730 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:29.126774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:29.126725 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" event={"ID":"0881f6d1-1062-471e-9d54-be23d9688330","Type":"ContainerStarted","Data":"850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1"} Nov 02 06:55:29.127151 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:29.127116 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:29.145112 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:29.145024 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:29.238115 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:29.238012 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" podStartSLOduration=2.314157944 podStartE2EDuration="3.237999438s" totalImagesPullingTime="923.841494ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:55:29.237182224 +0000 UTC m=+2305.685558978" watchObservedRunningTime="2026-11-02 06:55:29.237999438 +0000 UTC m=+2305.686376227" Nov 02 06:55:29.337675 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:29.337642 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:31.806536 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:31.806498 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:32.707715 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:55:32.707680 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:55:33.506097 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:33.496045 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:35.140928 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:35.140898 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:55:35.685419 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:35.685390 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:36.316419 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:36.316380 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:40.985850 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:40.985808 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:45.617102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:45.613850 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:47.421281 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:47.421239 5025 patch_prober.go:29] interesting pod/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 06:55:47.421742 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:47.421304 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 06:55:50.263851 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:50.263811 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:50.290758 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:50.290724 5025 generic.go:356] "Generic (PLEG): container finished" podID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerID="19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae" exitCode=0 Nov 02 06:55:50.290931 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:50.290801 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" event={"ID":"f67b2cd6-2ab3-4f88-a576-691f8837773e","Type":"ContainerDied","Data":"19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae"} Nov 02 06:55:50.291276 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:50.291261 5025 scope.go:122] "RemoveContainer" containerID="19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae" Nov 02 06:55:50.292760 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:50.292732 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:55:50.292815 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:50.292799 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:55:50.292846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:50.292829 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 06:55:51.313026 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:51.312985 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" event={"ID":"f67b2cd6-2ab3-4f88-a576-691f8837773e","Type":"ContainerStarted","Data":"6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393"} Nov 02 06:55:52.041150 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:52.041114 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:55:52.041349 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:52.041327 5025 patch_prober.go:29] interesting pod/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:55:52.041457 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:52.041382 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 06:55:53.041389 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:53.041350 5025 patch_prober.go:29] interesting pod/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:55:53.041875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:53.041407 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 06:55:54.041598 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:54.041557 5025 patch_prober.go:29] interesting pod/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:55:54.042088 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:54.041616 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 06:55:54.798057 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:54.798014 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:55:55.041660 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:55.041628 5025 patch_prober.go:29] interesting pod/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:55:55.042141 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:55.041685 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 06:55:56.041684 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:56.041643 5025 patch_prober.go:29] interesting pod/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" start-of-body= Nov 02 06:55:56.042135 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:56.041713 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.136:15021/healthz/ready\": dial tcp 10.133.0.136:15021: connect: connection refused" Nov 02 06:55:57.044667 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:57.044636 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:55:57.045085 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:57.044905 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:55:57.045596 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:57.045572 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 06:55:59.380870 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:55:59.378040 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:02.707883 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:56:02.707846 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:56:03.183252 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.183221 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz"] Nov 02 06:56:03.183440 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.183418 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" podUID="0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed" containerName="maas-api" containerID="cri-o://35351843a68a9702168ec92dd91b836f3d1ea380b428b8270098afde1630454a" gracePeriod=30 Nov 02 06:56:03.361217 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.361171 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:03.448442 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.448355 5025 generic.go:356] "Generic (PLEG): container finished" podID="0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed" containerID="35351843a68a9702168ec92dd91b836f3d1ea380b428b8270098afde1630454a" exitCode=0 Nov 02 06:56:03.448689 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.448496 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" event={"ID":"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed","Type":"ContainerDied","Data":"35351843a68a9702168ec92dd91b836f3d1ea380b428b8270098afde1630454a"} Nov 02 06:56:03.481981 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.481955 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:56:03.634909 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.634875 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-metrics-certs\") pod \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " Nov 02 06:56:03.635727 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.635700 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-kube-api-access-mczh8\" (UniqueName: \"kubernetes.io/projected/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-kube-api-access-mczh8\") pod \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " Nov 02 06:56:03.638215 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.638190 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-maas-api-tls\") pod \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " Nov 02 06:56:03.638509 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.638489 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-openshift-service-ca\") pod \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\" (UID: \"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed\") " Nov 02 06:56:03.640369 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.640323 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-openshift-service-ca" pod "0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed" (UID: "0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:56:03.641191 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.641159 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-metrics-certs" pod "0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed" (UID: "0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:56:03.643487 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.643455 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-kube-api-access-mczh8" pod "0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed" (UID: "0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed"). InnerVolumeSpecName "kube-api-access-mczh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:56:03.643691 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.643664 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-maas-api-tls" pod "0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed" (UID: "0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:56:03.745060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.744873 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:03.745608 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.745580 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:03.745744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.745612 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mczh8\" (UniqueName: \"kubernetes.io/projected/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-kube-api-access-mczh8\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:03.745744 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:03.745630 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:04.455227 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.455197 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" Nov 02 06:56:04.455227 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.455209 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz" event={"ID":"0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed","Type":"ContainerDied","Data":"6cf85a8851ce85d80ece160053fc15c3925bc0d4994241824ddb91586318e74d"} Nov 02 06:56:04.455456 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.455267 5025 scope.go:122] "RemoveContainer" containerID="35351843a68a9702168ec92dd91b836f3d1ea380b428b8270098afde1630454a" Nov 02 06:56:04.555056 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.555024 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85"] Nov 02 06:56:04.555637 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.555620 5025 cpu_manager.go:414] "containerMap: removing container" podUID="0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed" containerName="maas-api" Nov 02 06:56:04.555637 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.555637 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed" containerName="maas-api" Nov 02 06:56:04.555776 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.555713 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9c4a77-9cc7-4478-966e-e0762e9c5c97" containerName="istio-proxy" Nov 02 06:56:04.555776 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.555724 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed" containerName="maas-api" Nov 02 06:56:04.560323 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.560303 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.566788 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.566763 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85"] Nov 02 06:56:04.610477 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.610450 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2e9f81\"/\"kube-root-ca.crt\"" Nov 02 06:56:04.610579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.610519 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2e9f81\"/\"test-model-2e9f81-kserve-self-signed-certs\"" Nov 02 06:56:04.610579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.610545 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2e9f81\"/\"openshift-service-ca.crt\"" Nov 02 06:56:04.610579 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.610563 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2e9f81\"/\"default-dockercfg-wt2q2\"" Nov 02 06:56:04.668488 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.668448 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-model-cache\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.670607 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.670577 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-dshm\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.671351 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.671325 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-tmp-dir\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.671556 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.671537 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-home\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.671682 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.671666 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jl2d\" (UniqueName: \"kubernetes.io/projected/89055878-ba75-4384-8290-b3855b20d6af-kube-api-access-5jl2d\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.672411 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.672391 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89055878-ba75-4384-8290-b3855b20d6af-tls-certs\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.672611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.670386 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:04.691123 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.691068 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz"] Nov 02 06:56:04.693827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.693789 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-680b7b-5685b74796-wdxkz"] Nov 02 06:56:04.773543 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.773467 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-home\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.773543 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.773509 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jl2d\" (UniqueName: \"kubernetes.io/projected/89055878-ba75-4384-8290-b3855b20d6af-kube-api-access-5jl2d\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.774051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.773580 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89055878-ba75-4384-8290-b3855b20d6af-tls-certs\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.774051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.773611 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-model-cache\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.774051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.773829 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-dshm\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.774051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.773885 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-home\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.774051 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.774024 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-tmp-dir\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.774374 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.774190 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-model-cache\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.774430 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.774368 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-tmp-dir\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.775834 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.775808 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-dshm\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.776021 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.776004 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89055878-ba75-4384-8290-b3855b20d6af-tls-certs\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.831019 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.830993 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jl2d\" (UniqueName: \"kubernetes.io/projected/89055878-ba75-4384-8290-b3855b20d6af-kube-api-access-5jl2d\") pod \"test-model-2e9f81-kserve-775b67c557-vlx85\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:04.865339 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.865239 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:04.879425 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:04.879400 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:05.069520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:05.069438 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:05.247215 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:05.247188 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85"] Nov 02 06:56:05.248135 ip-10-0-130-34 kubenswrapper[5025]: W1102 06:56:05.248109 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89055878_ba75_4384_8290_b3855b20d6af.slice/crio-9de7f9cdae8e8f1296d95f730f9f5743427f4f1012156ed887cf3f0096998f14 WatchSource:0}: Error finding container 9de7f9cdae8e8f1296d95f730f9f5743427f4f1012156ed887cf3f0096998f14: Status 404 returned error can't find the container with id 9de7f9cdae8e8f1296d95f730f9f5743427f4f1012156ed887cf3f0096998f14 Nov 02 06:56:05.251042 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:05.251017 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:05.464548 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:05.464511 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" event={"ID":"89055878-ba75-4384-8290-b3855b20d6af","Type":"ContainerStarted","Data":"7e2a6bcc0da3d8eecd4dc9e685aa87241b8adc474eb41cdc1ae03257f183189e"} Nov 02 06:56:05.464739 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:05.464564 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" event={"ID":"89055878-ba75-4384-8290-b3855b20d6af","Type":"ContainerStarted","Data":"9de7f9cdae8e8f1296d95f730f9f5743427f4f1012156ed887cf3f0096998f14"} Nov 02 06:56:05.574999 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:05.574948 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" podStartSLOduration=1.5749344600000001 podStartE2EDuration="1.57493446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 06:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 06:56:05.57358549 +0000 UTC m=+2342.021962263" watchObservedRunningTime="2026-11-02 06:56:05.57493446 +0000 UTC m=+2342.023311278" Nov 02 06:56:05.668248 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:05.668207 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed" path="/var/lib/kubelet/pods/0dcd46fc-2776-4cbb-bb98-fee6a2e1e5ed/volumes" Nov 02 06:56:05.751678 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:05.751596 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:05.880314 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:05.880285 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:09.955161 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:09.955036 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:11.833317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:11.833280 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:12.155317 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:12.155219 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:12.158643 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:12.158617 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp"] Nov 02 06:56:12.158844 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:12.158824 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerName="istio-proxy" containerID="cri-o://e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31" gracePeriod=30 Nov 02 06:56:12.334096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:12.333101 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:14.570060 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:14.569840 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:14.880659 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:14.880558 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:14.892809 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:14.892785 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:15.529709 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:15.529676 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 06:56:17.441292 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.441259 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:56:17.513300 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.513220 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:17.533855 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.533821 5025 generic.go:356] "Generic (PLEG): container finished" podID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerID="e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31" exitCode=0 Nov 02 06:56:17.533999 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.533864 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" event={"ID":"a9765890-d47a-4002-a17a-a4e3afb2efe3","Type":"ContainerDied","Data":"e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31"} Nov 02 06:56:17.533999 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.533877 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" Nov 02 06:56:17.533999 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.533945 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp" event={"ID":"a9765890-d47a-4002-a17a-a4e3afb2efe3","Type":"ContainerDied","Data":"a9dd3f7931d9d03e6ff15f5da9a427a28f1ffefe5268b23fe0f121ba38a853e1"} Nov 02 06:56:17.533999 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.533973 5025 scope.go:122] "RemoveContainer" containerID="e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31" Nov 02 06:56:17.544502 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.544485 5025 scope.go:122] "RemoveContainer" containerID="e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31" Nov 02 06:56:17.544792 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:56:17.544768 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31\": container with ID starting with e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31 not found: ID does not exist" containerID="e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31" Nov 02 06:56:17.544891 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.544795 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31"} err="failed to get container status \"e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31\": rpc error: code = NotFound desc = could not find container \"e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31\": container with ID starting with e20480a678353aeae5af7eef08640b6e01bde20891aeb1c9021c6066b4d68c31 not found: ID does not exist" Nov 02 06:56:17.562245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562220 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-certs\") pod \"a9765890-d47a-4002-a17a-a4e3afb2efe3\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " Nov 02 06:56:17.562379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562269 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-credential-socket\") pod \"a9765890-d47a-4002-a17a-a4e3afb2efe3\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " Nov 02 06:56:17.562379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562332 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-token\" (UniqueName: \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-token\") pod \"a9765890-d47a-4002-a17a-a4e3afb2efe3\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " Nov 02 06:56:17.562379 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562354 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-data\") pod \"a9765890-d47a-4002-a17a-a4e3afb2efe3\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " Nov 02 06:56:17.562604 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562409 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9765890-d47a-4002-a17a-a4e3afb2efe3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9765890-d47a-4002-a17a-a4e3afb2efe3-istiod-ca-cert\") pod \"a9765890-d47a-4002-a17a-a4e3afb2efe3\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " Nov 02 06:56:17.562604 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562474 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-envoy\") pod \"a9765890-d47a-4002-a17a-a4e3afb2efe3\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " Nov 02 06:56:17.562604 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562511 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-kube-api-access-f6vpp\" (UniqueName: \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-kube-api-access-f6vpp\") pod \"a9765890-d47a-4002-a17a-a4e3afb2efe3\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " Nov 02 06:56:17.562604 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562591 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-socket\") pod \"a9765890-d47a-4002-a17a-a4e3afb2efe3\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " Nov 02 06:56:17.562604 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562595 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-certs" pod "a9765890-d47a-4002-a17a-a4e3afb2efe3" (UID: "a9765890-d47a-4002-a17a-a4e3afb2efe3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:56:17.562846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562766 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-credential-socket" pod "a9765890-d47a-4002-a17a-a4e3afb2efe3" (UID: "a9765890-d47a-4002-a17a-a4e3afb2efe3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:56:17.562846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562785 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-data" pod "a9765890-d47a-4002-a17a-a4e3afb2efe3" (UID: "a9765890-d47a-4002-a17a-a4e3afb2efe3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:56:17.562846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562815 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-podinfo\") pod \"a9765890-d47a-4002-a17a-a4e3afb2efe3\" (UID: \"a9765890-d47a-4002-a17a-a4e3afb2efe3\") " Nov 02 06:56:17.562846 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.562841 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9765890-d47a-4002-a17a-a4e3afb2efe3-istiod-ca-cert" pod "a9765890-d47a-4002-a17a-a4e3afb2efe3" (UID: "a9765890-d47a-4002-a17a-a4e3afb2efe3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:56:17.563274 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.563204 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-socket" pod "a9765890-d47a-4002-a17a-a4e3afb2efe3" (UID: "a9765890-d47a-4002-a17a-a4e3afb2efe3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:56:17.563438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.563404 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.563438 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.563433 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.563609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.563448 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.563609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.563461 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9765890-d47a-4002-a17a-a4e3afb2efe3-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.563609 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.563475 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.564722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.564692 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-token" pod "a9765890-d47a-4002-a17a-a4e3afb2efe3" (UID: "a9765890-d47a-4002-a17a-a4e3afb2efe3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:56:17.564961 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.564931 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-envoy" pod "a9765890-d47a-4002-a17a-a4e3afb2efe3" (UID: "a9765890-d47a-4002-a17a-a4e3afb2efe3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:56:17.565180 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.565152 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-kube-api-access-f6vpp" pod "a9765890-d47a-4002-a17a-a4e3afb2efe3" (UID: "a9765890-d47a-4002-a17a-a4e3afb2efe3"). InnerVolumeSpecName "kube-api-access-f6vpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:56:17.565823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.565803 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-podinfo" pod "a9765890-d47a-4002-a17a-a4e3afb2efe3" (UID: "a9765890-d47a-4002-a17a-a4e3afb2efe3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:56:17.664987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.664947 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.664987 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.664987 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.665210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.665005 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6vpp\" (UniqueName: \"kubernetes.io/projected/a9765890-d47a-4002-a17a-a4e3afb2efe3-kube-api-access-f6vpp\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.665210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.665019 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9765890-d47a-4002-a17a-a4e3afb2efe3-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:56:17.721030 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.720994 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:17.944237 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.944203 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp"] Nov 02 06:56:17.948012 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:17.947990 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-680b7b-openshift-default-9cdb9d8f9-2l2kp"] Nov 02 06:56:19.107100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:19.105119 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:19.668930 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:19.668896 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" path="/var/lib/kubelet/pods/a9765890-d47a-4002-a17a-a4e3afb2efe3/volumes" Nov 02 06:56:23.630130 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:23.626987 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:26.858191 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:26.858143 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:28.265519 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:28.265484 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:30.417876 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:30.417838 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:32.706827 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:56:32.706788 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:56:33.165593 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:33.162876 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:37.819103 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:37.818480 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:42.483450 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:42.483408 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:43.587106 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:43.585197 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:43.774380 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:43.774344 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:47.084848 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:47.082690 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:51.646209 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:51.646169 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:56.162121 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:56.162026 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:59.081023 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:59.080986 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:56:59.267974 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:56:59.267934 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:00.689439 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:00.689400 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:02.705840 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:57:02.705804 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:57:05.311893 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:05.311852 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:09.980477 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:09.980442 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:14.490906 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:14.490869 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:19.099986 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:19.099921 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:22.177094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:22.177031 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:22.375993 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:22.375962 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:23.784099 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:23.782723 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:24.803139 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:24.803105 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2"] Nov 02 06:57:24.803540 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:24.803319 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" containerID="cri-o://813d4c91244c821f4102e068ecc8154a2a2c94a5aa9088b06a4a3b8ce52d181f" gracePeriod=30 Nov 02 06:57:25.074774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:25.074699 5025 patch_prober.go:29] interesting pod/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:57:25.074774 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:25.074761 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:57:27.193510 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:27.193472 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:28.307255 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:28.307220 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:29.953882 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:29.953850 5025 generic.go:356] "Generic (PLEG): container finished" podID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerID="813d4c91244c821f4102e068ecc8154a2a2c94a5aa9088b06a4a3b8ce52d181f" exitCode=0 Nov 02 06:57:29.954292 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:29.953918 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" event={"ID":"4885f2f1-b780-46b0-90f9-91f624acc4f9","Type":"ContainerDied","Data":"813d4c91244c821f4102e068ecc8154a2a2c94a5aa9088b06a4a3b8ce52d181f"} Nov 02 06:57:30.059731 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.059699 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:57:30.180874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.180843 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-envoy\") pod \"4885f2f1-b780-46b0-90f9-91f624acc4f9\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " Nov 02 06:57:30.181059 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.180882 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-data\") pod \"4885f2f1-b780-46b0-90f9-91f624acc4f9\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " Nov 02 06:57:30.181059 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.180947 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4885f2f1-b780-46b0-90f9-91f624acc4f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4885f2f1-b780-46b0-90f9-91f624acc4f9-istiod-ca-cert\") pod \"4885f2f1-b780-46b0-90f9-91f624acc4f9\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " Nov 02 06:57:30.181059 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.180973 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-credential-socket\") pod \"4885f2f1-b780-46b0-90f9-91f624acc4f9\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " Nov 02 06:57:30.181261 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.181063 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-kube-api-access-pf4vn\" (UniqueName: \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-kube-api-access-pf4vn\") pod \"4885f2f1-b780-46b0-90f9-91f624acc4f9\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " Nov 02 06:57:30.181261 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.181131 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-socket\") pod \"4885f2f1-b780-46b0-90f9-91f624acc4f9\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " Nov 02 06:57:30.181261 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.181179 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-token\" (UniqueName: \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-token\") pod \"4885f2f1-b780-46b0-90f9-91f624acc4f9\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " Nov 02 06:57:30.181261 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.181205 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-data" pod "4885f2f1-b780-46b0-90f9-91f624acc4f9" (UID: "4885f2f1-b780-46b0-90f9-91f624acc4f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:30.181261 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.181215 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-podinfo\") pod \"4885f2f1-b780-46b0-90f9-91f624acc4f9\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " Nov 02 06:57:30.181528 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.181286 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-certs\") pod \"4885f2f1-b780-46b0-90f9-91f624acc4f9\" (UID: \"4885f2f1-b780-46b0-90f9-91f624acc4f9\") " Nov 02 06:57:30.181528 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.181353 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-credential-socket" pod "4885f2f1-b780-46b0-90f9-91f624acc4f9" (UID: "4885f2f1-b780-46b0-90f9-91f624acc4f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:30.181639 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.181547 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4885f2f1-b780-46b0-90f9-91f624acc4f9-istiod-ca-cert" pod "4885f2f1-b780-46b0-90f9-91f624acc4f9" (UID: "4885f2f1-b780-46b0-90f9-91f624acc4f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:57:30.181753 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.181729 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-certs" pod "4885f2f1-b780-46b0-90f9-91f624acc4f9" (UID: "4885f2f1-b780-46b0-90f9-91f624acc4f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:30.181873 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.181850 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-socket" pod "4885f2f1-b780-46b0-90f9-91f624acc4f9" (UID: "4885f2f1-b780-46b0-90f9-91f624acc4f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:30.182007 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.181983 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:30.182007 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.182008 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:30.182216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.182023 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:30.182216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.182038 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4885f2f1-b780-46b0-90f9-91f624acc4f9-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:30.182216 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.182052 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:30.183085 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.183044 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-envoy" pod "4885f2f1-b780-46b0-90f9-91f624acc4f9" (UID: "4885f2f1-b780-46b0-90f9-91f624acc4f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:30.183367 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.183346 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-token" pod "4885f2f1-b780-46b0-90f9-91f624acc4f9" (UID: "4885f2f1-b780-46b0-90f9-91f624acc4f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:57:30.183367 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.183348 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-podinfo" pod "4885f2f1-b780-46b0-90f9-91f624acc4f9" (UID: "4885f2f1-b780-46b0-90f9-91f624acc4f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:57:30.183684 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.183663 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-kube-api-access-pf4vn" pod "4885f2f1-b780-46b0-90f9-91f624acc4f9" (UID: "4885f2f1-b780-46b0-90f9-91f624acc4f9"). InnerVolumeSpecName "kube-api-access-pf4vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:57:30.282954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.282923 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:30.282954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.282949 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:30.282954 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.282962 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4885f2f1-b780-46b0-90f9-91f624acc4f9-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:30.283210 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.282970 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf4vn\" (UniqueName: \"kubernetes.io/projected/4885f2f1-b780-46b0-90f9-91f624acc4f9-kube-api-access-pf4vn\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:30.961416 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.961380 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" Nov 02 06:57:30.961416 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.961401 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2" event={"ID":"4885f2f1-b780-46b0-90f9-91f624acc4f9","Type":"ContainerDied","Data":"b98bb22b38650daa1c4da03bcadce0e54c45b298db238b8879199255157f6e34"} Nov 02 06:57:30.961874 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:30.961462 5025 scope.go:122] "RemoveContainer" containerID="813d4c91244c821f4102e068ecc8154a2a2c94a5aa9088b06a4a3b8ce52d181f" Nov 02 06:57:31.165470 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:31.165438 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2"] Nov 02 06:57:31.212741 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:31.212672 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-cdfce9-openshift-default-6c9cf4b7b7-wnhq2"] Nov 02 06:57:31.461689 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:31.461616 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:31.664117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:31.664068 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:31.668542 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:31.668520 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" path="/var/lib/kubelet/pods/4885f2f1-b780-46b0-90f9-91f624acc4f9/volumes" Nov 02 06:57:32.707108 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:57:32.707052 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:57:32.862112 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:32.862017 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:37.405325 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:37.405281 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:40.094793 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:40.094760 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq"] Nov 02 06:57:40.095265 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:40.094954 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" containerID="cri-o://8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da" gracePeriod=30 Nov 02 06:57:41.272770 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:41.272727 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86"] Nov 02 06:57:41.273299 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:41.273007 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" podUID="7c654f68-3867-449f-8f59-626b947927b4" containerName="istio-proxy" containerID="cri-o://a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e" gracePeriod=30 Nov 02 06:57:42.067869 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:42.067827 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:45.365391 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.365329 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:57:45.463715 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.463685 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-credential-socket\") pod \"60924270-4daa-44e7-a026-cc44a28d77eb\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " Nov 02 06:57:45.463875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.463778 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-istio-token\" (UniqueName: \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-istio-token\") pod \"60924270-4daa-44e7-a026-cc44a28d77eb\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " Nov 02 06:57:45.463875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.463805 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-certs\") pod \"60924270-4daa-44e7-a026-cc44a28d77eb\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " Nov 02 06:57:45.463875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.463826 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60924270-4daa-44e7-a026-cc44a28d77eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60924270-4daa-44e7-a026-cc44a28d77eb-istio-podinfo\") pod \"60924270-4daa-44e7-a026-cc44a28d77eb\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " Nov 02 06:57:45.463875 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.463866 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-data\") pod \"60924270-4daa-44e7-a026-cc44a28d77eb\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " Nov 02 06:57:45.464069 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.463949 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-socket\") pod \"60924270-4daa-44e7-a026-cc44a28d77eb\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " Nov 02 06:57:45.464069 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.463981 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60924270-4daa-44e7-a026-cc44a28d77eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60924270-4daa-44e7-a026-cc44a28d77eb-istiod-ca-cert\") pod \"60924270-4daa-44e7-a026-cc44a28d77eb\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " Nov 02 06:57:45.464069 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.464043 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-kube-api-access-xqlrg\" (UniqueName: \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-kube-api-access-xqlrg\") pod \"60924270-4daa-44e7-a026-cc44a28d77eb\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " Nov 02 06:57:45.464069 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.464089 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-envoy\") pod \"60924270-4daa-44e7-a026-cc44a28d77eb\" (UID: \"60924270-4daa-44e7-a026-cc44a28d77eb\") " Nov 02 06:57:45.464288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.463947 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-credential-socket" pod "60924270-4daa-44e7-a026-cc44a28d77eb" (UID: "60924270-4daa-44e7-a026-cc44a28d77eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:45.464288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.464217 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-socket" pod "60924270-4daa-44e7-a026-cc44a28d77eb" (UID: "60924270-4daa-44e7-a026-cc44a28d77eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:45.464288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.464111 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-certs" pod "60924270-4daa-44e7-a026-cc44a28d77eb" (UID: "60924270-4daa-44e7-a026-cc44a28d77eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:45.464288 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.464201 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-data" pod "60924270-4daa-44e7-a026-cc44a28d77eb" (UID: "60924270-4daa-44e7-a026-cc44a28d77eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:45.464488 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.464419 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60924270-4daa-44e7-a026-cc44a28d77eb-istiod-ca-cert" pod "60924270-4daa-44e7-a026-cc44a28d77eb" (UID: "60924270-4daa-44e7-a026-cc44a28d77eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:57:45.464637 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.464617 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60924270-4daa-44e7-a026-cc44a28d77eb-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:45.464707 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.464640 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:45.464707 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.464656 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:45.464707 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.464669 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:45.464707 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.464683 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:45.466104 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.466052 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-istio-token" pod "60924270-4daa-44e7-a026-cc44a28d77eb" (UID: "60924270-4daa-44e7-a026-cc44a28d77eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:57:45.466219 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.466102 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-kube-api-access-xqlrg" pod "60924270-4daa-44e7-a026-cc44a28d77eb" (UID: "60924270-4daa-44e7-a026-cc44a28d77eb"). InnerVolumeSpecName "kube-api-access-xqlrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:57:45.466219 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.466102 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60924270-4daa-44e7-a026-cc44a28d77eb-istio-podinfo" pod "60924270-4daa-44e7-a026-cc44a28d77eb" (UID: "60924270-4daa-44e7-a026-cc44a28d77eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:57:45.466294 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.466221 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-envoy" pod "60924270-4daa-44e7-a026-cc44a28d77eb" (UID: "60924270-4daa-44e7-a026-cc44a28d77eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:45.565615 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.565573 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60924270-4daa-44e7-a026-cc44a28d77eb-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:45.565615 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.565608 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqlrg\" (UniqueName: \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-kube-api-access-xqlrg\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:45.565832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.565625 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60924270-4daa-44e7-a026-cc44a28d77eb-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:45.565832 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:45.565639 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60924270-4daa-44e7-a026-cc44a28d77eb-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:46.043476 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.043397 5025 generic.go:356] "Generic (PLEG): container finished" podID="60924270-4daa-44e7-a026-cc44a28d77eb" containerID="8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da" exitCode=0 Nov 02 06:57:46.043476 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.043447 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" Nov 02 06:57:46.043656 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.043487 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" event={"ID":"60924270-4daa-44e7-a026-cc44a28d77eb","Type":"ContainerDied","Data":"8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da"} Nov 02 06:57:46.043656 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.043543 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq" event={"ID":"60924270-4daa-44e7-a026-cc44a28d77eb","Type":"ContainerDied","Data":"32acf9b5e1a2dc0c36966963373400f9b940b05aefbc4428a68ff6430fd1ee4f"} Nov 02 06:57:46.043656 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.043565 5025 scope.go:122] "RemoveContainer" containerID="8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da" Nov 02 06:57:46.053102 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.053063 5025 scope.go:122] "RemoveContainer" containerID="75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf" Nov 02 06:57:46.061191 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.061174 5025 scope.go:122] "RemoveContainer" containerID="8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da" Nov 02 06:57:46.061485 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:57:46.061467 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da\": container with ID starting with 8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da not found: ID does not exist" containerID="8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da" Nov 02 06:57:46.061532 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.061495 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da"} err="failed to get container status \"8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da\": rpc error: code = NotFound desc = could not find container \"8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da\": container with ID starting with 8a9411662cdd14c135ae4e7a0803d84e22841321675f484653cbafb0611682da not found: ID does not exist" Nov 02 06:57:46.061532 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.061513 5025 scope.go:122] "RemoveContainer" containerID="75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf" Nov 02 06:57:46.061754 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:57:46.061733 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf\": container with ID starting with 75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf not found: ID does not exist" containerID="75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf" Nov 02 06:57:46.061812 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.061763 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf"} err="failed to get container status \"75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf\": rpc error: code = NotFound desc = could not find container \"75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf\": container with ID starting with 75fb5dca41d30f273627ca672767e588494551f713c454e896097086fedcb8cf not found: ID does not exist" Nov 02 06:57:46.201132 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.201096 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq"] Nov 02 06:57:46.248499 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.248466 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-581608-openshift-default-cd7c896df-g9qsq"] Nov 02 06:57:46.530696 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.530663 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:57:46.578421 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.578351 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-envoy\") pod \"7c654f68-3867-449f-8f59-626b947927b4\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " Nov 02 06:57:46.578568 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.578482 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-credential-socket\") pod \"7c654f68-3867-449f-8f59-626b947927b4\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " Nov 02 06:57:46.578568 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.578515 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c654f68-3867-449f-8f59-626b947927b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c654f68-3867-449f-8f59-626b947927b4-istiod-ca-cert\") pod \"7c654f68-3867-449f-8f59-626b947927b4\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " Nov 02 06:57:46.578568 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.578538 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c654f68-3867-449f-8f59-626b947927b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c654f68-3867-449f-8f59-626b947927b4-istio-podinfo\") pod \"7c654f68-3867-449f-8f59-626b947927b4\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " Nov 02 06:57:46.578722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.578576 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-kube-api-access-7d7ll\" (UniqueName: \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-kube-api-access-7d7ll\") pod \"7c654f68-3867-449f-8f59-626b947927b4\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " Nov 02 06:57:46.578722 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.578686 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-istio-token\" (UniqueName: \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-istio-token\") pod \"7c654f68-3867-449f-8f59-626b947927b4\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " Nov 02 06:57:46.578823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.578721 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-certs\") pod \"7c654f68-3867-449f-8f59-626b947927b4\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " Nov 02 06:57:46.578823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.578745 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-credential-socket" pod "7c654f68-3867-449f-8f59-626b947927b4" (UID: "7c654f68-3867-449f-8f59-626b947927b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:46.578823 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.578812 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-socket\") pod \"7c654f68-3867-449f-8f59-626b947927b4\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " Nov 02 06:57:46.578975 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.578877 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c654f68-3867-449f-8f59-626b947927b4-istiod-ca-cert" pod "7c654f68-3867-449f-8f59-626b947927b4" (UID: "7c654f68-3867-449f-8f59-626b947927b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:57:46.578975 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.578883 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-data\") pod \"7c654f68-3867-449f-8f59-626b947927b4\" (UID: \"7c654f68-3867-449f-8f59-626b947927b4\") " Nov 02 06:57:46.579164 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.579120 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-data" pod "7c654f68-3867-449f-8f59-626b947927b4" (UID: "7c654f68-3867-449f-8f59-626b947927b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:46.579301 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.579257 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-certs" pod "7c654f68-3867-449f-8f59-626b947927b4" (UID: "7c654f68-3867-449f-8f59-626b947927b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:46.579301 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.579274 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-socket" pod "7c654f68-3867-449f-8f59-626b947927b4" (UID: "7c654f68-3867-449f-8f59-626b947927b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:46.579558 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.579542 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:46.579645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.579562 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:46.579645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.579576 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:46.579645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.579589 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c654f68-3867-449f-8f59-626b947927b4-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:46.579645 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.579602 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:46.580821 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.580795 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-istio-token" pod "7c654f68-3867-449f-8f59-626b947927b4" (UID: "7c654f68-3867-449f-8f59-626b947927b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:57:46.580941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.580842 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-kube-api-access-7d7ll" pod "7c654f68-3867-449f-8f59-626b947927b4" (UID: "7c654f68-3867-449f-8f59-626b947927b4"). InnerVolumeSpecName "kube-api-access-7d7ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:57:46.581002 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.580936 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-envoy" pod "7c654f68-3867-449f-8f59-626b947927b4" (UID: "7c654f68-3867-449f-8f59-626b947927b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 06:57:46.581117 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.581067 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c654f68-3867-449f-8f59-626b947927b4-istio-podinfo" pod "7c654f68-3867-449f-8f59-626b947927b4" (UID: "7c654f68-3867-449f-8f59-626b947927b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 06:57:46.681094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.681048 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c654f68-3867-449f-8f59-626b947927b4-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:46.681094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.681092 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c654f68-3867-449f-8f59-626b947927b4-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:46.681094 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.681103 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d7ll\" (UniqueName: \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-kube-api-access-7d7ll\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:46.681386 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.681111 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c654f68-3867-449f-8f59-626b947927b4-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:57:46.681932 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:46.681902 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:47.050435 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:47.050398 5025 generic.go:356] "Generic (PLEG): container finished" podID="7c654f68-3867-449f-8f59-626b947927b4" containerID="a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e" exitCode=0 Nov 02 06:57:47.050643 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:47.050450 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" Nov 02 06:57:47.050643 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:47.050480 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" event={"ID":"7c654f68-3867-449f-8f59-626b947927b4","Type":"ContainerDied","Data":"a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e"} Nov 02 06:57:47.050643 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:47.050534 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86" event={"ID":"7c654f68-3867-449f-8f59-626b947927b4","Type":"ContainerDied","Data":"4e48a379bc2ca9d6666b8908537d19493b25286510510f50e1b5e124fc13bfd4"} Nov 02 06:57:47.050643 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:47.050561 5025 scope.go:122] "RemoveContainer" containerID="a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e" Nov 02 06:57:47.060245 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:47.060228 5025 scope.go:122] "RemoveContainer" containerID="a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e" Nov 02 06:57:47.060492 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:57:47.060476 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e\": container with ID starting with a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e not found: ID does not exist" containerID="a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e" Nov 02 06:57:47.060540 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:47.060506 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e"} err="failed to get container status \"a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e\": rpc error: code = NotFound desc = could not find container \"a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e\": container with ID starting with a3ea355f23594df15f613c2f6b8be75a7067fa72ff54e7d90d15c537897af85e not found: ID does not exist" Nov 02 06:57:47.208333 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:47.208298 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86"] Nov 02 06:57:47.255262 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:47.255235 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-6ff85e-openshift-default-698585659c-grf86"] Nov 02 06:57:47.667614 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:47.667577 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" path="/var/lib/kubelet/pods/60924270-4daa-44e7-a026-cc44a28d77eb/volumes" Nov 02 06:57:47.668105 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:47.668062 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c654f68-3867-449f-8f59-626b947927b4" path="/var/lib/kubelet/pods/7c654f68-3867-449f-8f59-626b947927b4/volumes" Nov 02 06:57:51.271520 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:51.271482 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:55.809917 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:55.809879 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:57:59.778456 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:57:59.778422 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:01.241348 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:01.241316 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:01.553938 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:01.553856 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:01.744113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:01.744068 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:02.224989 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:02.224954 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:02.707403 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:58:02.707370 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:58:06.378971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:06.378926 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:08.329322 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:08.329290 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:08.515484 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:08.515441 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:08.705672 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:08.705599 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:08.904467 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:08.904412 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:11.165820 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:11.165547 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:14.386740 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:14.386707 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:14.622401 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:14.622373 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:16.086246 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:16.086202 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:20.693444 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:20.693401 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:21.179066 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:21.179026 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:21.367027 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:21.366955 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:25.311611 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:25.311573 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:27.538660 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:27.538623 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:29.886771 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:29.886719 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:32.706639 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:58:32.706604 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:58:34.500548 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:34.500503 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:39.073971 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:39.073941 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:43.090145 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:43.090114 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:43.271789 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:43.271752 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:43.659556 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:43.659516 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:48.213054 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:48.213008 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:52.770468 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:52.770433 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:57.311306 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:57.311264 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:57.503213 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:57.503176 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:59.605526 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:59.605495 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:58:59.783289 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:58:59.783259 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:02.000929 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:02.000892 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:02.706732 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:59:02.706691 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:59:06.580886 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:06.580853 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:11.182490 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:11.182455 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:15.547792 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:15.547546 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:15.799703 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:15.799619 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:20.467133 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:20.467054 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:25.073168 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:25.073135 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:27.850232 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:27.850199 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:29.685465 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:29.685411 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:31.613156 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:31.613118 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:32.707742 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:59:32.707703 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 06:59:34.400756 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:34.400718 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:39.002941 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:39.002837 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:39.613406 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:39.613370 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:43.575235 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:43.575056 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:47.438390 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.438351 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn"] Nov 02 06:59:47.438969 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.438572 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" podUID="0881f6d1-1062-471e-9d54-be23d9688330" containerName="maas-api" containerID="cri-o://850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1" gracePeriod=30 Nov 02 06:59:47.584345 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.584311 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:47.715393 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.715365 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:59:47.781693 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.781659 5025 generic.go:356] "Generic (PLEG): container finished" podID="0881f6d1-1062-471e-9d54-be23d9688330" containerID="850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1" exitCode=0 Nov 02 06:59:47.781856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.781706 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" Nov 02 06:59:47.781856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.781760 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" event={"ID":"0881f6d1-1062-471e-9d54-be23d9688330","Type":"ContainerDied","Data":"850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1"} Nov 02 06:59:47.781856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.781810 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn" event={"ID":"0881f6d1-1062-471e-9d54-be23d9688330","Type":"ContainerDied","Data":"7bf116ac438ed14eda2428b761c8af72e08a02b4e04df30774388952974d39b1"} Nov 02 06:59:47.781856 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.781832 5025 scope.go:122] "RemoveContainer" containerID="850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1" Nov 02 06:59:47.791042 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.791008 5025 scope.go:122] "RemoveContainer" containerID="850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1" Nov 02 06:59:47.791322 ip-10-0-130-34 kubenswrapper[5025]: E1102 06:59:47.791302 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1\": container with ID starting with 850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1 not found: ID does not exist" containerID="850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1" Nov 02 06:59:47.791411 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.791332 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1"} err="failed to get container status \"850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1\": rpc error: code = NotFound desc = could not find container \"850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1\": container with ID starting with 850ec755dee7275cb7a10bf292b2c629bee25fb938189afcb4471efcab762ea1 not found: ID does not exist" Nov 02 06:59:47.827096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.825444 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-maas-api-tls\") pod \"0881f6d1-1062-471e-9d54-be23d9688330\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " Nov 02 06:59:47.827096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.825521 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0881f6d1-1062-471e-9d54-be23d9688330-kube-api-access-hccz8\" (UniqueName: \"kubernetes.io/projected/0881f6d1-1062-471e-9d54-be23d9688330-kube-api-access-hccz8\") pod \"0881f6d1-1062-471e-9d54-be23d9688330\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " Nov 02 06:59:47.827096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.825620 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-metrics-certs\") pod \"0881f6d1-1062-471e-9d54-be23d9688330\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " Nov 02 06:59:47.827096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.825650 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0881f6d1-1062-471e-9d54-be23d9688330-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0881f6d1-1062-471e-9d54-be23d9688330-openshift-service-ca\") pod \"0881f6d1-1062-471e-9d54-be23d9688330\" (UID: \"0881f6d1-1062-471e-9d54-be23d9688330\") " Nov 02 06:59:47.827096 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.826825 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0881f6d1-1062-471e-9d54-be23d9688330-openshift-service-ca" pod "0881f6d1-1062-471e-9d54-be23d9688330" (UID: "0881f6d1-1062-471e-9d54-be23d9688330"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 06:59:47.842100 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.841818 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-metrics-certs" pod "0881f6d1-1062-471e-9d54-be23d9688330" (UID: "0881f6d1-1062-471e-9d54-be23d9688330"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:59:47.854827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.852574 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0881f6d1-1062-471e-9d54-be23d9688330-kube-api-access-hccz8" pod "0881f6d1-1062-471e-9d54-be23d9688330" (UID: "0881f6d1-1062-471e-9d54-be23d9688330"). InnerVolumeSpecName "kube-api-access-hccz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 06:59:47.854827 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.854232 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-maas-api-tls" pod "0881f6d1-1062-471e-9d54-be23d9688330" (UID: "0881f6d1-1062-471e-9d54-be23d9688330"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 06:59:47.927918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.927887 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hccz8\" (UniqueName: \"kubernetes.io/projected/0881f6d1-1062-471e-9d54-be23d9688330-kube-api-access-hccz8\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:47.927918 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.927920 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:47.928122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.927930 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0881f6d1-1062-471e-9d54-be23d9688330-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:47.928122 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:47.927939 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0881f6d1-1062-471e-9d54-be23d9688330-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 06:59:48.198704 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:48.198670 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn"] Nov 02 06:59:48.202374 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:48.202351 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e6fde6-58889947d8-ljvdn"] Nov 02 06:59:49.055346 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:49.055314 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:49.261388 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:49.261358 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:49.461181 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:49.461144 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:49.668797 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:49.668763 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0881f6d1-1062-471e-9d54-be23d9688330" path="/var/lib/kubelet/pods/0881f6d1-1062-471e-9d54-be23d9688330/volumes" Nov 02 06:59:50.018113 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:50.018064 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:54.062472 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:54.062438 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:55.848466 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:55.848435 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:56.028468 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:56.028436 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:56.103517 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:56.103441 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq"] Nov 02 06:59:56.103682 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:56.103656 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" containerID="cri-o://6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393" gracePeriod=30 Nov 02 06:59:56.221158 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:56.216398 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:56.585665 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:56.585583 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:56.678162 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:56.678127 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85"] Nov 02 06:59:56.780190 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:56.780157 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:57.047861 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:57.047823 5025 patch_prober.go:29] interesting pod/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 06:59:57.048359 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:57.047895 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 06:59:59.010298 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:59.010254 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 06:59:59.575230 ip-10-0-130-34 kubenswrapper[5025]: I1102 06:59:59.575194 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:00.156863 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.156822 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6"] Nov 02 07:00:00.157533 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157512 5025 cpu_manager.go:414] "containerMap: removing container" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerName="istio-proxy" Nov 02 07:00:00.157629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157535 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerName="istio-proxy" Nov 02 07:00:00.157629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157547 5025 cpu_manager.go:414] "containerMap: removing container" podUID="7c654f68-3867-449f-8f59-626b947927b4" containerName="istio-proxy" Nov 02 07:00:00.157629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157555 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c654f68-3867-449f-8f59-626b947927b4" containerName="istio-proxy" Nov 02 07:00:00.157629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157583 5025 cpu_manager.go:414] "containerMap: removing container" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" Nov 02 07:00:00.157629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157591 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" Nov 02 07:00:00.157629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157623 5025 cpu_manager.go:414] "containerMap: removing container" podUID="0881f6d1-1062-471e-9d54-be23d9688330" containerName="maas-api" Nov 02 07:00:00.157629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157631 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0881f6d1-1062-471e-9d54-be23d9688330" containerName="maas-api" Nov 02 07:00:00.157976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157649 5025 cpu_manager.go:414] "containerMap: removing container" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" Nov 02 07:00:00.157976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157656 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" Nov 02 07:00:00.157976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157666 5025 cpu_manager.go:414] "containerMap: removing container" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" Nov 02 07:00:00.157976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157673 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" Nov 02 07:00:00.157976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157789 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" Nov 02 07:00:00.157976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157807 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="0881f6d1-1062-471e-9d54-be23d9688330" containerName="maas-api" Nov 02 07:00:00.157976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157817 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="4885f2f1-b780-46b0-90f9-91f624acc4f9" containerName="istio-proxy" Nov 02 07:00:00.157976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157825 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="60924270-4daa-44e7-a026-cc44a28d77eb" containerName="istio-proxy" Nov 02 07:00:00.157976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157838 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9765890-d47a-4002-a17a-a4e3afb2efe3" containerName="istio-proxy" Nov 02 07:00:00.157976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.157850 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c654f68-3867-449f-8f59-626b947927b4" containerName="istio-proxy" Nov 02 07:00:00.161164 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.161142 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" Nov 02 07:00:00.165418 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.165390 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx"] Nov 02 07:00:00.168975 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.168957 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" Nov 02 07:00:00.173103 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.169635 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64"] Nov 02 07:00:00.179727 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.179706 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" Nov 02 07:00:00.182063 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.182041 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ac4602-feb7-4f17-8634-c31db023e71a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6\" (UID: \"83ac4602-feb7-4f17-8634-c31db023e71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" Nov 02 07:00:00.182418 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.182394 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvfr7\" (UniqueName: \"kubernetes.io/projected/83ac4602-feb7-4f17-8634-c31db023e71a-kube-api-access-bvfr7\") pod \"maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6\" (UID: \"83ac4602-feb7-4f17-8634-c31db023e71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" Nov 02 07:00:00.204369 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.204346 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n"] Nov 02 07:00:00.208132 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.208113 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25"] Nov 02 07:00:00.208236 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.208209 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" Nov 02 07:00:00.211543 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.211525 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" Nov 02 07:00:00.214364 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.214347 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:00:00.252024 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.251993 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6"] Nov 02 07:00:00.283670 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.283635 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvfr7\" (UniqueName: \"kubernetes.io/projected/83ac4602-feb7-4f17-8634-c31db023e71a-kube-api-access-bvfr7\") pod \"maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6\" (UID: \"83ac4602-feb7-4f17-8634-c31db023e71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" Nov 02 07:00:00.283854 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.283709 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ac4602-feb7-4f17-8634-c31db023e71a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6\" (UID: \"83ac4602-feb7-4f17-8634-c31db023e71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" Nov 02 07:00:00.283854 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.283752 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmls2\" (UniqueName: \"kubernetes.io/projected/2ddb53c7-b78a-42ff-9e2a-2661872d1890-kube-api-access-rmls2\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25\" (UID: \"2ddb53c7-b78a-42ff-9e2a-2661872d1890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" Nov 02 07:00:00.283854 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.283777 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbtm7\" (UniqueName: \"kubernetes.io/projected/d6abd316-2c0e-43ec-9c51-9600f833b021-kube-api-access-lbtm7\") pod \"maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n\" (UID: \"d6abd316-2c0e-43ec-9c51-9600f833b021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" Nov 02 07:00:00.283854 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.283801 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddb53c7-b78a-42ff-9e2a-2661872d1890-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25\" (UID: \"2ddb53c7-b78a-42ff-9e2a-2661872d1890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" Nov 02 07:00:00.284024 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.283959 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmpfz\" (UniqueName: \"kubernetes.io/projected/300f0119-015c-45eb-8758-f8c7ee065e7f-kube-api-access-wmpfz\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx\" (UID: \"300f0119-015c-45eb-8758-f8c7ee065e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" Nov 02 07:00:00.284094 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.284068 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/300f0119-015c-45eb-8758-f8c7ee065e7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx\" (UID: \"300f0119-015c-45eb-8758-f8c7ee065e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" Nov 02 07:00:00.284134 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.284113 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64\" (UID: \"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" Nov 02 07:00:00.284177 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.284164 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx49t\" (UniqueName: \"kubernetes.io/projected/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-kube-api-access-mx49t\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64\" (UID: \"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" Nov 02 07:00:00.284227 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.284214 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6abd316-2c0e-43ec-9c51-9600f833b021-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n\" (UID: \"d6abd316-2c0e-43ec-9c51-9600f833b021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" Nov 02 07:00:00.284686 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.284662 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ac4602-feb7-4f17-8634-c31db023e71a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6\" (UID: \"83ac4602-feb7-4f17-8634-c31db023e71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" Nov 02 07:00:00.295147 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.295107 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64"] Nov 02 07:00:00.298145 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.298119 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx"] Nov 02 07:00:00.301057 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.299755 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25"] Nov 02 07:00:00.302478 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.302457 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n"] Nov 02 07:00:00.339148 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.339113 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvfr7\" (UniqueName: \"kubernetes.io/projected/83ac4602-feb7-4f17-8634-c31db023e71a-kube-api-access-bvfr7\") pod \"maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6\" (UID: \"83ac4602-feb7-4f17-8634-c31db023e71a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" Nov 02 07:00:00.385100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.385045 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6abd316-2c0e-43ec-9c51-9600f833b021-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n\" (UID: \"d6abd316-2c0e-43ec-9c51-9600f833b021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" Nov 02 07:00:00.385275 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.385255 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmls2\" (UniqueName: \"kubernetes.io/projected/2ddb53c7-b78a-42ff-9e2a-2661872d1890-kube-api-access-rmls2\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25\" (UID: \"2ddb53c7-b78a-42ff-9e2a-2661872d1890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" Nov 02 07:00:00.385362 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.385287 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbtm7\" (UniqueName: \"kubernetes.io/projected/d6abd316-2c0e-43ec-9c51-9600f833b021-kube-api-access-lbtm7\") pod \"maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n\" (UID: \"d6abd316-2c0e-43ec-9c51-9600f833b021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" Nov 02 07:00:00.385362 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.385311 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddb53c7-b78a-42ff-9e2a-2661872d1890-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25\" (UID: \"2ddb53c7-b78a-42ff-9e2a-2661872d1890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" Nov 02 07:00:00.385478 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.385451 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmpfz\" (UniqueName: \"kubernetes.io/projected/300f0119-015c-45eb-8758-f8c7ee065e7f-kube-api-access-wmpfz\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx\" (UID: \"300f0119-015c-45eb-8758-f8c7ee065e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" Nov 02 07:00:00.385557 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.385529 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/300f0119-015c-45eb-8758-f8c7ee065e7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx\" (UID: \"300f0119-015c-45eb-8758-f8c7ee065e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" Nov 02 07:00:00.385621 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.385567 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64\" (UID: \"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" Nov 02 07:00:00.385621 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.385597 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx49t\" (UniqueName: \"kubernetes.io/projected/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-kube-api-access-mx49t\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64\" (UID: \"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" Nov 02 07:00:00.385767 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.385745 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6abd316-2c0e-43ec-9c51-9600f833b021-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n\" (UID: \"d6abd316-2c0e-43ec-9c51-9600f833b021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" Nov 02 07:00:00.386112 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.386071 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddb53c7-b78a-42ff-9e2a-2661872d1890-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25\" (UID: \"2ddb53c7-b78a-42ff-9e2a-2661872d1890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" Nov 02 07:00:00.386192 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.386173 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64\" (UID: \"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" Nov 02 07:00:00.386241 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.386176 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/300f0119-015c-45eb-8758-f8c7ee065e7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx\" (UID: \"300f0119-015c-45eb-8758-f8c7ee065e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" Nov 02 07:00:00.439545 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.439519 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmpfz\" (UniqueName: \"kubernetes.io/projected/300f0119-015c-45eb-8758-f8c7ee065e7f-kube-api-access-wmpfz\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx\" (UID: \"300f0119-015c-45eb-8758-f8c7ee065e7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" Nov 02 07:00:00.439641 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.439523 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx49t\" (UniqueName: \"kubernetes.io/projected/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-kube-api-access-mx49t\") pod \"maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64\" (UID: \"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" Nov 02 07:00:00.439689 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.439678 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbtm7\" (UniqueName: \"kubernetes.io/projected/d6abd316-2c0e-43ec-9c51-9600f833b021-kube-api-access-lbtm7\") pod \"maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n\" (UID: \"d6abd316-2c0e-43ec-9c51-9600f833b021\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" Nov 02 07:00:00.439822 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.439798 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmls2\" (UniqueName: \"kubernetes.io/projected/2ddb53c7-b78a-42ff-9e2a-2661872d1890-kube-api-access-rmls2\") pod \"maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25\" (UID: \"2ddb53c7-b78a-42ff-9e2a-2661872d1890\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" Nov 02 07:00:00.481062 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.481040 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" Nov 02 07:00:00.486474 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.486446 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" Nov 02 07:00:00.495125 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.495099 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" Nov 02 07:00:00.521864 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.521824 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" Nov 02 07:00:00.536766 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.530317 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" Nov 02 07:00:00.867322 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.867295 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6"] Nov 02 07:00:00.868004 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:00:00.867978 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ac4602_feb7_4f17_8634_c31db023e71a.slice/crio-7e58c9bde2457dca118dc17916b28a45c137cc26e21a152490a884c443f95ebb WatchSource:0}: Error finding container 7e58c9bde2457dca118dc17916b28a45c137cc26e21a152490a884c443f95ebb: Status 404 returned error can't find the container with id 7e58c9bde2457dca118dc17916b28a45c137cc26e21a152490a884c443f95ebb Nov 02 07:00:00.914873 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:00:00.914841 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31aa8fb2_6c55_40a2_80c5_7b12b10e1ae6.slice/crio-f40d038a3fabaf912eb64cb0b24c3103c8e478fdfdfa3f9946375eca29ddc4fa WatchSource:0}: Error finding container f40d038a3fabaf912eb64cb0b24c3103c8e478fdfdfa3f9946375eca29ddc4fa: Status 404 returned error can't find the container with id f40d038a3fabaf912eb64cb0b24c3103c8e478fdfdfa3f9946375eca29ddc4fa Nov 02 07:00:00.915016 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.914885 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64"] Nov 02 07:00:00.916877 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:00:00.916856 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod300f0119_015c_45eb_8758_f8c7ee065e7f.slice/crio-87331e5dbc32d3cc9e80f247818a5171460829fb5391b318024007e1ff6db308 WatchSource:0}: Error finding container 87331e5dbc32d3cc9e80f247818a5171460829fb5391b318024007e1ff6db308: Status 404 returned error can't find the container with id 87331e5dbc32d3cc9e80f247818a5171460829fb5391b318024007e1ff6db308 Nov 02 07:00:00.917114 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.917093 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx"] Nov 02 07:00:00.961874 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.961845 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25"] Nov 02 07:00:00.963141 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:00:00.963047 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ddb53c7_b78a_42ff_9e2a_2661872d1890.slice/crio-56f00b37f1f2ad5534cc76d475cd6ad92ac1216715882e14f43e1e1857d3f10c WatchSource:0}: Error finding container 56f00b37f1f2ad5534cc76d475cd6ad92ac1216715882e14f43e1e1857d3f10c: Status 404 returned error can't find the container with id 56f00b37f1f2ad5534cc76d475cd6ad92ac1216715882e14f43e1e1857d3f10c Nov 02 07:00:00.965010 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:00.964988 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n"] Nov 02 07:00:00.965216 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:00:00.965197 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6abd316_2c0e_43ec_9c51_9600f833b021.slice/crio-da0c2362eef375b2f5a7c15b88e15460bf11281a3aba97ca2df357837928666a WatchSource:0}: Error finding container da0c2362eef375b2f5a7c15b88e15460bf11281a3aba97ca2df357837928666a: Status 404 returned error can't find the container with id da0c2362eef375b2f5a7c15b88e15460bf11281a3aba97ca2df357837928666a Nov 02 07:00:01.337174 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.337151 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 07:00:01.398971 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.398944 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-credential-socket\") pod \"f67b2cd6-2ab3-4f88-a576-691f8837773e\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " Nov 02 07:00:01.399097 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.398989 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-socket\") pod \"f67b2cd6-2ab3-4f88-a576-691f8837773e\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " Nov 02 07:00:01.399097 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399013 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-podinfo\") pod \"f67b2cd6-2ab3-4f88-a576-691f8837773e\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " Nov 02 07:00:01.399097 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399039 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-envoy\") pod \"f67b2cd6-2ab3-4f88-a576-691f8837773e\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " Nov 02 07:00:01.399097 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399097 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-data\") pod \"f67b2cd6-2ab3-4f88-a576-691f8837773e\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " Nov 02 07:00:01.399331 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399116 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f67b2cd6-2ab3-4f88-a576-691f8837773e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67b2cd6-2ab3-4f88-a576-691f8837773e-istiod-ca-cert\") pod \"f67b2cd6-2ab3-4f88-a576-691f8837773e\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " Nov 02 07:00:01.399331 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399230 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-certs\") pod \"f67b2cd6-2ab3-4f88-a576-691f8837773e\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " Nov 02 07:00:01.399331 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399259 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-credential-socket" pod "f67b2cd6-2ab3-4f88-a576-691f8837773e" (UID: "f67b2cd6-2ab3-4f88-a576-691f8837773e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:01.399331 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399275 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-socket" pod "f67b2cd6-2ab3-4f88-a576-691f8837773e" (UID: "f67b2cd6-2ab3-4f88-a576-691f8837773e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:01.399593 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399351 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-kube-api-access-klfpb\" (UniqueName: \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-kube-api-access-klfpb\") pod \"f67b2cd6-2ab3-4f88-a576-691f8837773e\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " Nov 02 07:00:01.399593 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399410 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-data" pod "f67b2cd6-2ab3-4f88-a576-691f8837773e" (UID: "f67b2cd6-2ab3-4f88-a576-691f8837773e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:01.399593 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399432 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-token\" (UniqueName: \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-token\") pod \"f67b2cd6-2ab3-4f88-a576-691f8837773e\" (UID: \"f67b2cd6-2ab3-4f88-a576-691f8837773e\") " Nov 02 07:00:01.399593 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399485 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-certs" pod "f67b2cd6-2ab3-4f88-a576-691f8837773e" (UID: "f67b2cd6-2ab3-4f88-a576-691f8837773e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:01.399804 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.399783 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f67b2cd6-2ab3-4f88-a576-691f8837773e-istiod-ca-cert" pod "f67b2cd6-2ab3-4f88-a576-691f8837773e" (UID: "f67b2cd6-2ab3-4f88-a576-691f8837773e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:01.400031 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.400012 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.400151 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.400034 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f67b2cd6-2ab3-4f88-a576-691f8837773e-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.400151 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.400049 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.400151 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.400062 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.400151 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.400089 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.401270 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.401242 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-podinfo" pod "f67b2cd6-2ab3-4f88-a576-691f8837773e" (UID: "f67b2cd6-2ab3-4f88-a576-691f8837773e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:00:01.401398 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.401379 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-kube-api-access-klfpb" pod "f67b2cd6-2ab3-4f88-a576-691f8837773e" (UID: "f67b2cd6-2ab3-4f88-a576-691f8837773e"). InnerVolumeSpecName "kube-api-access-klfpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:01.401458 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.401430 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-envoy" pod "f67b2cd6-2ab3-4f88-a576-691f8837773e" (UID: "f67b2cd6-2ab3-4f88-a576-691f8837773e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:01.401576 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.401559 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-token" pod "f67b2cd6-2ab3-4f88-a576-691f8837773e" (UID: "f67b2cd6-2ab3-4f88-a576-691f8837773e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:01.501069 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.500991 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klfpb\" (UniqueName: \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-kube-api-access-klfpb\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.501069 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.501021 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.501069 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.501031 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.501069 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.501041 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f67b2cd6-2ab3-4f88-a576-691f8837773e-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:01.797187 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.797161 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:01.857349 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.857315 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" event={"ID":"d6abd316-2c0e-43ec-9c51-9600f833b021","Type":"ContainerStarted","Data":"dc942e62a353c6cdae46849cb22314c2048000c83fd8d3af4edd81c8a00f384f"} Nov 02 07:00:01.857495 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.857373 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" event={"ID":"d6abd316-2c0e-43ec-9c51-9600f833b021","Type":"ContainerStarted","Data":"da0c2362eef375b2f5a7c15b88e15460bf11281a3aba97ca2df357837928666a"} Nov 02 07:00:01.858818 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.858794 5025 generic.go:356] "Generic (PLEG): container finished" podID="83ac4602-feb7-4f17-8634-c31db023e71a" containerID="500cb2d03a950ee407ae6620b6f4177509a745e44bb98dcff9ad28386943f5f7" exitCode=60 Nov 02 07:00:01.858921 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.858864 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" event={"ID":"83ac4602-feb7-4f17-8634-c31db023e71a","Type":"ContainerDied","Data":"500cb2d03a950ee407ae6620b6f4177509a745e44bb98dcff9ad28386943f5f7"} Nov 02 07:00:01.858921 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.858900 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" event={"ID":"83ac4602-feb7-4f17-8634-c31db023e71a","Type":"ContainerStarted","Data":"7e58c9bde2457dca118dc17916b28a45c137cc26e21a152490a884c443f95ebb"} Nov 02 07:00:01.859137 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.859120 5025 scope.go:122] "RemoveContainer" containerID="500cb2d03a950ee407ae6620b6f4177509a745e44bb98dcff9ad28386943f5f7" Nov 02 07:00:01.860683 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.860591 5025 generic.go:356] "Generic (PLEG): container finished" podID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerID="6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393" exitCode=0 Nov 02 07:00:01.860683 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.860643 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" Nov 02 07:00:01.860683 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.860669 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" event={"ID":"f67b2cd6-2ab3-4f88-a576-691f8837773e","Type":"ContainerDied","Data":"6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393"} Nov 02 07:00:01.860855 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.860715 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq" event={"ID":"f67b2cd6-2ab3-4f88-a576-691f8837773e","Type":"ContainerDied","Data":"b2624ef5ec61b7a33a41efed6470e8b1e268c7ca9fc8293ea324be33d5ec5eed"} Nov 02 07:00:01.860855 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.860742 5025 scope.go:122] "RemoveContainer" containerID="6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393" Nov 02 07:00:01.866484 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.866209 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" event={"ID":"300f0119-015c-45eb-8758-f8c7ee065e7f","Type":"ContainerStarted","Data":"fdae1552fa36b9bff33b1e517f464bc50055b0bb31613082a2b2254891976135"} Nov 02 07:00:01.866484 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.866302 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" event={"ID":"300f0119-015c-45eb-8758-f8c7ee065e7f","Type":"ContainerStarted","Data":"87331e5dbc32d3cc9e80f247818a5171460829fb5391b318024007e1ff6db308"} Nov 02 07:00:01.869608 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.869578 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" event={"ID":"2ddb53c7-b78a-42ff-9e2a-2661872d1890","Type":"ContainerStarted","Data":"2111a4fc0ddcd56168e4cd8ae44f128d8e672c1680d1529cb6369eb491c49bb4"} Nov 02 07:00:01.869698 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.869639 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" event={"ID":"2ddb53c7-b78a-42ff-9e2a-2661872d1890","Type":"ContainerStarted","Data":"56f00b37f1f2ad5534cc76d475cd6ad92ac1216715882e14f43e1e1857d3f10c"} Nov 02 07:00:01.872981 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.872957 5025 generic.go:356] "Generic (PLEG): container finished" podID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerID="63bbfe5c5c136b0a93af81cf221076ff9657caecb61b7d3f2344bab1c928226e" exitCode=60 Nov 02 07:00:01.873130 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.872987 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" event={"ID":"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6","Type":"ContainerDied","Data":"63bbfe5c5c136b0a93af81cf221076ff9657caecb61b7d3f2344bab1c928226e"} Nov 02 07:00:01.873130 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.873015 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" event={"ID":"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6","Type":"ContainerStarted","Data":"f40d038a3fabaf912eb64cb0b24c3103c8e478fdfdfa3f9946375eca29ddc4fa"} Nov 02 07:00:01.873309 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.873289 5025 scope.go:122] "RemoveContainer" containerID="63bbfe5c5c136b0a93af81cf221076ff9657caecb61b7d3f2344bab1c928226e" Nov 02 07:00:01.879947 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.879929 5025 scope.go:122] "RemoveContainer" containerID="19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae" Nov 02 07:00:01.892745 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.892725 5025 scope.go:122] "RemoveContainer" containerID="6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393" Nov 02 07:00:01.893106 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:01.892984 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393\": container with ID starting with 6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393 not found: ID does not exist" containerID="6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393" Nov 02 07:00:01.893106 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.893011 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393"} err="failed to get container status \"6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393\": rpc error: code = NotFound desc = could not find container \"6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393\": container with ID starting with 6e47387c1f51c8b2710363bed379f4d6da08ff774660e03e23994770f86e6393 not found: ID does not exist" Nov 02 07:00:01.893106 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.893028 5025 scope.go:122] "RemoveContainer" containerID="19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae" Nov 02 07:00:01.893310 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:01.893290 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae\": container with ID starting with 19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae not found: ID does not exist" containerID="19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae" Nov 02 07:00:01.893347 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.893317 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae"} err="failed to get container status \"19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae\": rpc error: code = NotFound desc = could not find container \"19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae\": container with ID starting with 19457d6c4c1375bcf9961126cc2e5fb9aceeba010786dfdaaa6d3017fc36c3ae not found: ID does not exist" Nov 02 07:00:01.972043 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:01.971995 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" podStartSLOduration=1.971982414 podStartE2EDuration="1.971982414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:01.970062581 +0000 UTC m=+2578.418439354" watchObservedRunningTime="2026-11-02 07:00:01.971982414 +0000 UTC m=+2578.420359207" Nov 02 07:00:02.081041 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.080990 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" podStartSLOduration=2.080976179 podStartE2EDuration="2.080976179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:02.079234203 +0000 UTC m=+2578.527610985" watchObservedRunningTime="2026-11-02 07:00:02.080976179 +0000 UTC m=+2578.529352933" Nov 02 07:00:02.186577 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.186545 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq"] Nov 02 07:00:02.192510 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.192486 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e6fde6-openshift-default-6649ddc6c-kbvhq"] Nov 02 07:00:02.300121 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.300002 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" podStartSLOduration=2.29998774 podStartE2EDuration="2.29998774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:02.297815722 +0000 UTC m=+2578.746192479" watchObservedRunningTime="2026-11-02 07:00:02.29998774 +0000 UTC m=+2578.748364495" Nov 02 07:00:02.706229 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:02.706186 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:00:02.881270 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.881179 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" event={"ID":"83ac4602-feb7-4f17-8634-c31db023e71a","Type":"ContainerStarted","Data":"359a77a3abeb31ce7dde6e6590b9c8df460c82df160533f6f2c77c25e6be7b76"} Nov 02 07:00:02.883448 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.883421 5025 generic.go:356] "Generic (PLEG): container finished" podID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerID="fdae1552fa36b9bff33b1e517f464bc50055b0bb31613082a2b2254891976135" exitCode=60 Nov 02 07:00:02.883603 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.883505 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" event={"ID":"300f0119-015c-45eb-8758-f8c7ee065e7f","Type":"ContainerDied","Data":"fdae1552fa36b9bff33b1e517f464bc50055b0bb31613082a2b2254891976135"} Nov 02 07:00:02.883741 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.883727 5025 scope.go:122] "RemoveContainer" containerID="fdae1552fa36b9bff33b1e517f464bc50055b0bb31613082a2b2254891976135" Nov 02 07:00:02.885708 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.885684 5025 generic.go:356] "Generic (PLEG): container finished" podID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerID="2111a4fc0ddcd56168e4cd8ae44f128d8e672c1680d1529cb6369eb491c49bb4" exitCode=60 Nov 02 07:00:02.885835 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.885766 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" event={"ID":"2ddb53c7-b78a-42ff-9e2a-2661872d1890","Type":"ContainerDied","Data":"2111a4fc0ddcd56168e4cd8ae44f128d8e672c1680d1529cb6369eb491c49bb4"} Nov 02 07:00:02.885995 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.885979 5025 scope.go:122] "RemoveContainer" containerID="2111a4fc0ddcd56168e4cd8ae44f128d8e672c1680d1529cb6369eb491c49bb4" Nov 02 07:00:02.888618 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.888140 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" event={"ID":"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6","Type":"ContainerStarted","Data":"2f4b99a4fd3e8f50a01a440f3b54744ceb81e1accb1f52936a31af5f51093948"} Nov 02 07:00:02.889956 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.889935 5025 generic.go:356] "Generic (PLEG): container finished" podID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerID="dc942e62a353c6cdae46849cb22314c2048000c83fd8d3af4edd81c8a00f384f" exitCode=60 Nov 02 07:00:02.890064 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.889968 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" event={"ID":"d6abd316-2c0e-43ec-9c51-9600f833b021","Type":"ContainerDied","Data":"dc942e62a353c6cdae46849cb22314c2048000c83fd8d3af4edd81c8a00f384f"} Nov 02 07:00:02.893111 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.890217 5025 scope.go:122] "RemoveContainer" containerID="dc942e62a353c6cdae46849cb22314c2048000c83fd8d3af4edd81c8a00f384f" Nov 02 07:00:02.991466 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:02.991422 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" podStartSLOduration=2.991409783 podStartE2EDuration="2.991409783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:02.990612894 +0000 UTC m=+2579.438989668" watchObservedRunningTime="2026-11-02 07:00:02.991409783 +0000 UTC m=+2579.439786582" Nov 02 07:00:03.099680 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.099634 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" podStartSLOduration=3.099621219 podStartE2EDuration="3.099621219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:00:03.099199017 +0000 UTC m=+2579.547575771" watchObservedRunningTime="2026-11-02 07:00:03.099621219 +0000 UTC m=+2579.547998010" Nov 02 07:00:03.669347 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.669308 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" path="/var/lib/kubelet/pods/f67b2cd6-2ab3-4f88-a576-691f8837773e/volumes" Nov 02 07:00:03.897159 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.897121 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" event={"ID":"300f0119-015c-45eb-8758-f8c7ee065e7f","Type":"ContainerStarted","Data":"16a9d20f47f4515f4637b87cd0d6aca2423b7813909e274ec324d2ac8ef0b827"} Nov 02 07:00:03.899038 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.899009 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" event={"ID":"2ddb53c7-b78a-42ff-9e2a-2661872d1890","Type":"ContainerStarted","Data":"f3b191ce255c207a6c1a89ee15179d6ab8835fb07a8930dd15fbe4d03d9c3299"} Nov 02 07:00:03.900760 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.900737 5025 generic.go:356] "Generic (PLEG): container finished" podID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerID="2f4b99a4fd3e8f50a01a440f3b54744ceb81e1accb1f52936a31af5f51093948" exitCode=60 Nov 02 07:00:03.900845 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.900812 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" event={"ID":"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6","Type":"ContainerDied","Data":"2f4b99a4fd3e8f50a01a440f3b54744ceb81e1accb1f52936a31af5f51093948"} Nov 02 07:00:03.900901 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.900857 5025 scope.go:122] "RemoveContainer" containerID="63bbfe5c5c136b0a93af81cf221076ff9657caecb61b7d3f2344bab1c928226e" Nov 02 07:00:03.900968 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.900950 5025 scope.go:122] "RemoveContainer" containerID="2f4b99a4fd3e8f50a01a440f3b54744ceb81e1accb1f52936a31af5f51093948" Nov 02 07:00:03.901240 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:03.901214 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64_odh-ai-gateway-infra(31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" Nov 02 07:00:03.902705 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.902687 5025 generic.go:356] "Generic (PLEG): container finished" podID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerID="196b241bce321aaee34bf964b6d04f93165f7147fa57039437a7ad8859dadff6" exitCode=60 Nov 02 07:00:03.902790 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.902771 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" event={"ID":"d6abd316-2c0e-43ec-9c51-9600f833b021","Type":"ContainerDied","Data":"196b241bce321aaee34bf964b6d04f93165f7147fa57039437a7ad8859dadff6"} Nov 02 07:00:03.903006 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.902988 5025 scope.go:122] "RemoveContainer" containerID="196b241bce321aaee34bf964b6d04f93165f7147fa57039437a7ad8859dadff6" Nov 02 07:00:03.903239 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:03.903219 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n_odh-ai-gateway-infra(d6abd316-2c0e-43ec-9c51-9600f833b021)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" Nov 02 07:00:03.904589 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.904561 5025 generic.go:356] "Generic (PLEG): container finished" podID="83ac4602-feb7-4f17-8634-c31db023e71a" containerID="359a77a3abeb31ce7dde6e6590b9c8df460c82df160533f6f2c77c25e6be7b76" exitCode=60 Nov 02 07:00:03.904756 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.904604 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" event={"ID":"83ac4602-feb7-4f17-8634-c31db023e71a","Type":"ContainerDied","Data":"359a77a3abeb31ce7dde6e6590b9c8df460c82df160533f6f2c77c25e6be7b76"} Nov 02 07:00:03.904844 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.904827 5025 scope.go:122] "RemoveContainer" containerID="359a77a3abeb31ce7dde6e6590b9c8df460c82df160533f6f2c77c25e6be7b76" Nov 02 07:00:03.905037 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:03.905018 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6_odh-ai-gateway-infra(83ac4602-feb7-4f17-8634-c31db023e71a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" Nov 02 07:00:03.912420 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.912402 5025 scope.go:122] "RemoveContainer" containerID="dc942e62a353c6cdae46849cb22314c2048000c83fd8d3af4edd81c8a00f384f" Nov 02 07:00:03.924226 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:03.924208 5025 scope.go:122] "RemoveContainer" containerID="500cb2d03a950ee407ae6620b6f4177509a745e44bb98dcff9ad28386943f5f7" Nov 02 07:00:04.112062 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.110466 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:04.911694 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.911605 5025 scope.go:122] "RemoveContainer" containerID="196b241bce321aaee34bf964b6d04f93165f7147fa57039437a7ad8859dadff6" Nov 02 07:00:04.912147 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:04.911933 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n_odh-ai-gateway-infra(d6abd316-2c0e-43ec-9c51-9600f833b021)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" Nov 02 07:00:04.913060 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.913037 5025 scope.go:122] "RemoveContainer" containerID="359a77a3abeb31ce7dde6e6590b9c8df460c82df160533f6f2c77c25e6be7b76" Nov 02 07:00:04.913361 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:04.913338 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6_odh-ai-gateway-infra(83ac4602-feb7-4f17-8634-c31db023e71a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" Nov 02 07:00:04.914551 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.914528 5025 generic.go:356] "Generic (PLEG): container finished" podID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerID="16a9d20f47f4515f4637b87cd0d6aca2423b7813909e274ec324d2ac8ef0b827" exitCode=60 Nov 02 07:00:04.914650 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.914606 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" event={"ID":"300f0119-015c-45eb-8758-f8c7ee065e7f","Type":"ContainerDied","Data":"16a9d20f47f4515f4637b87cd0d6aca2423b7813909e274ec324d2ac8ef0b827"} Nov 02 07:00:04.914714 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.914701 5025 scope.go:122] "RemoveContainer" containerID="fdae1552fa36b9bff33b1e517f464bc50055b0bb31613082a2b2254891976135" Nov 02 07:00:04.914817 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.914800 5025 scope.go:122] "RemoveContainer" containerID="16a9d20f47f4515f4637b87cd0d6aca2423b7813909e274ec324d2ac8ef0b827" Nov 02 07:00:04.915050 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:04.915031 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx_odh-ai-gateway-infra(300f0119-015c-45eb-8758-f8c7ee065e7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" Nov 02 07:00:04.916425 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.916404 5025 generic.go:356] "Generic (PLEG): container finished" podID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerID="f3b191ce255c207a6c1a89ee15179d6ab8835fb07a8930dd15fbe4d03d9c3299" exitCode=60 Nov 02 07:00:04.916515 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.916486 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" event={"ID":"2ddb53c7-b78a-42ff-9e2a-2661872d1890","Type":"ContainerDied","Data":"f3b191ce255c207a6c1a89ee15179d6ab8835fb07a8930dd15fbe4d03d9c3299"} Nov 02 07:00:04.916670 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.916655 5025 scope.go:122] "RemoveContainer" containerID="f3b191ce255c207a6c1a89ee15179d6ab8835fb07a8930dd15fbe4d03d9c3299" Nov 02 07:00:04.916916 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:04.916894 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25_odh-ai-gateway-infra(2ddb53c7-b78a-42ff-9e2a-2661872d1890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" Nov 02 07:00:04.918096 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.918055 5025 scope.go:122] "RemoveContainer" containerID="2f4b99a4fd3e8f50a01a440f3b54744ceb81e1accb1f52936a31af5f51093948" Nov 02 07:00:04.918267 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:04.918251 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64_odh-ai-gateway-infra(31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" Nov 02 07:00:04.927429 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:04.927412 5025 scope.go:122] "RemoveContainer" containerID="2111a4fc0ddcd56168e4cd8ae44f128d8e672c1680d1529cb6369eb491c49bb4" Nov 02 07:00:05.925238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:05.925208 5025 scope.go:122] "RemoveContainer" containerID="16a9d20f47f4515f4637b87cd0d6aca2423b7813909e274ec324d2ac8ef0b827" Nov 02 07:00:05.925638 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:05.925543 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx_odh-ai-gateway-infra(300f0119-015c-45eb-8758-f8c7ee065e7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" Nov 02 07:00:05.926757 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:05.926735 5025 scope.go:122] "RemoveContainer" containerID="f3b191ce255c207a6c1a89ee15179d6ab8835fb07a8930dd15fbe4d03d9c3299" Nov 02 07:00:05.926969 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:05.926953 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25_odh-ai-gateway-infra(2ddb53c7-b78a-42ff-9e2a-2661872d1890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" Nov 02 07:00:08.228660 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:08.228628 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2"] Nov 02 07:00:08.621380 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:08.621300 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:08.803210 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:08.803176 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:09.183105 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:09.176975 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:10.945148 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:10.944695 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:11.134734 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:11.134698 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:11.698260 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:11.698224 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" podUID="89055878-ba75-4384-8290-b3855b20d6af" containerName="main" containerID="cri-o://7e2a6bcc0da3d8eecd4dc9e685aa87241b8adc474eb41cdc1ae03257f183189e" gracePeriod=45 Nov 02 07:00:11.985010 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:11.984972 5025 generic.go:356] "Generic (PLEG): container finished" podID="89055878-ba75-4384-8290-b3855b20d6af" containerID="7e2a6bcc0da3d8eecd4dc9e685aa87241b8adc474eb41cdc1ae03257f183189e" exitCode=0 Nov 02 07:00:11.985382 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:11.985046 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" event={"ID":"89055878-ba75-4384-8290-b3855b20d6af","Type":"ContainerDied","Data":"7e2a6bcc0da3d8eecd4dc9e685aa87241b8adc474eb41cdc1ae03257f183189e"} Nov 02 07:00:12.008976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.008949 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 07:00:12.180792 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.178132 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-tmp-dir\") pod \"89055878-ba75-4384-8290-b3855b20d6af\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " Nov 02 07:00:12.193138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.178460 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-tmp-dir" pod "89055878-ba75-4384-8290-b3855b20d6af" (UID: "89055878-ba75-4384-8290-b3855b20d6af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:12.193138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.192712 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-home\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-home\") pod \"89055878-ba75-4384-8290-b3855b20d6af\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " Nov 02 07:00:12.193138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.192761 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89055878-ba75-4384-8290-b3855b20d6af-tls-certs\" (UniqueName: \"kubernetes.io/secret/89055878-ba75-4384-8290-b3855b20d6af-tls-certs\") pod \"89055878-ba75-4384-8290-b3855b20d6af\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " Nov 02 07:00:12.193138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.192826 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89055878-ba75-4384-8290-b3855b20d6af-kube-api-access-5jl2d\" (UniqueName: \"kubernetes.io/projected/89055878-ba75-4384-8290-b3855b20d6af-kube-api-access-5jl2d\") pod \"89055878-ba75-4384-8290-b3855b20d6af\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " Nov 02 07:00:12.193138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.192920 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-dshm\") pod \"89055878-ba75-4384-8290-b3855b20d6af\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " Nov 02 07:00:12.193138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.192961 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-model-cache\") pod \"89055878-ba75-4384-8290-b3855b20d6af\" (UID: \"89055878-ba75-4384-8290-b3855b20d6af\") " Nov 02 07:00:12.193613 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.193439 5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-tmp-dir\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:12.199698 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.193703 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-model-cache" pod "89055878-ba75-4384-8290-b3855b20d6af" (UID: "89055878-ba75-4384-8290-b3855b20d6af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:12.199698 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.194009 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-home" pod "89055878-ba75-4384-8290-b3855b20d6af" (UID: "89055878-ba75-4384-8290-b3855b20d6af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:12.220320 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.220283 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-dshm" pod "89055878-ba75-4384-8290-b3855b20d6af" (UID: "89055878-ba75-4384-8290-b3855b20d6af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:12.220515 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.220496 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89055878-ba75-4384-8290-b3855b20d6af-kube-api-access-5jl2d" pod "89055878-ba75-4384-8290-b3855b20d6af" (UID: "89055878-ba75-4384-8290-b3855b20d6af"). InnerVolumeSpecName "kube-api-access-5jl2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:12.220642 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.220623 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89055878-ba75-4384-8290-b3855b20d6af-tls-certs" pod "89055878-ba75-4384-8290-b3855b20d6af" (UID: "89055878-ba75-4384-8290-b3855b20d6af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:00:12.294604 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.294508 5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-model-cache\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:12.294604 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.294550 5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-home\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:12.294604 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.294559 5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89055878-ba75-4384-8290-b3855b20d6af-tls-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:12.294604 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.294569 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jl2d\" (UniqueName: \"kubernetes.io/projected/89055878-ba75-4384-8290-b3855b20d6af-kube-api-access-5jl2d\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:12.294604 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.294577 5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89055878-ba75-4384-8290-b3855b20d6af-dshm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:12.992511 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.992473 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" Nov 02 07:00:12.992953 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.992510 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85" event={"ID":"89055878-ba75-4384-8290-b3855b20d6af","Type":"ContainerDied","Data":"9de7f9cdae8e8f1296d95f730f9f5743427f4f1012156ed887cf3f0096998f14"} Nov 02 07:00:12.992953 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:12.992569 5025 scope.go:122] "RemoveContainer" containerID="7e2a6bcc0da3d8eecd4dc9e685aa87241b8adc474eb41cdc1ae03257f183189e" Nov 02 07:00:13.152729 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:13.152699 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85"] Nov 02 07:00:13.159924 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:13.159897 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2e9f81/test-model-2e9f81-kserve-775b67c557-vlx85"] Nov 02 07:00:13.223305 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:13.223276 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:13.229313 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:13.229267 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" podUID="49f36379-dc5e-434c-84be-4266642f39ca" containerName="payload-processing" containerID="cri-o://34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b" gracePeriod=25 Nov 02 07:00:13.401798 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:13.401724 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:13.668359 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:13.668318 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89055878-ba75-4384-8290-b3855b20d6af" path="/var/lib/kubelet/pods/89055878-ba75-4384-8290-b3855b20d6af/volumes" Nov 02 07:00:13.996770 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:13.996673 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:15.662577 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:15.662541 5025 scope.go:122] "RemoveContainer" containerID="2f4b99a4fd3e8f50a01a440f3b54744ceb81e1accb1f52936a31af5f51093948" Nov 02 07:00:16.031252 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:16.031222 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" event={"ID":"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6","Type":"ContainerStarted","Data":"2722d78a6ab9121a4aedd1559b666dceb00e7d1927242eeaf780f35b41d48574"} Nov 02 07:00:16.662261 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:16.662232 5025 scope.go:122] "RemoveContainer" containerID="16a9d20f47f4515f4637b87cd0d6aca2423b7813909e274ec324d2ac8ef0b827" Nov 02 07:00:16.662394 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:16.662355 5025 scope.go:122] "RemoveContainer" containerID="196b241bce321aaee34bf964b6d04f93165f7147fa57039437a7ad8859dadff6" Nov 02 07:00:16.662452 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:16.662417 5025 scope.go:122] "RemoveContainer" containerID="359a77a3abeb31ce7dde6e6590b9c8df460c82df160533f6f2c77c25e6be7b76" Nov 02 07:00:17.039802 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:17.039770 5025 generic.go:356] "Generic (PLEG): container finished" podID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerID="2722d78a6ab9121a4aedd1559b666dceb00e7d1927242eeaf780f35b41d48574" exitCode=60 Nov 02 07:00:17.040294 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:17.039845 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" event={"ID":"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6","Type":"ContainerDied","Data":"2722d78a6ab9121a4aedd1559b666dceb00e7d1927242eeaf780f35b41d48574"} Nov 02 07:00:17.040294 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:17.039907 5025 scope.go:122] "RemoveContainer" containerID="2f4b99a4fd3e8f50a01a440f3b54744ceb81e1accb1f52936a31af5f51093948" Nov 02 07:00:17.040294 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:17.040259 5025 scope.go:122] "RemoveContainer" containerID="2722d78a6ab9121a4aedd1559b666dceb00e7d1927242eeaf780f35b41d48574" Nov 02 07:00:17.040580 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:17.040558 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64_odh-ai-gateway-infra(31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" Nov 02 07:00:17.106237 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:17.106199 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" podUID="49f36379-dc5e-434c-84be-4266642f39ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:00:17.157346 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:17.157279 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64"] Nov 02 07:00:17.661847 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:17.661821 5025 scope.go:122] "RemoveContainer" containerID="f3b191ce255c207a6c1a89ee15179d6ab8835fb07a8930dd15fbe4d03d9c3299" Nov 02 07:00:18.048256 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.048229 5025 generic.go:356] "Generic (PLEG): container finished" podID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerID="99b308e177eed017b8b6b744abc766e2c12b54409ff7234590b28781b66ca7fe" exitCode=60 Nov 02 07:00:18.048638 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.048296 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" event={"ID":"d6abd316-2c0e-43ec-9c51-9600f833b021","Type":"ContainerDied","Data":"99b308e177eed017b8b6b744abc766e2c12b54409ff7234590b28781b66ca7fe"} Nov 02 07:00:18.048638 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.048368 5025 scope.go:122] "RemoveContainer" containerID="196b241bce321aaee34bf964b6d04f93165f7147fa57039437a7ad8859dadff6" Nov 02 07:00:18.048876 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.048850 5025 scope.go:122] "RemoveContainer" containerID="99b308e177eed017b8b6b744abc766e2c12b54409ff7234590b28781b66ca7fe" Nov 02 07:00:18.049342 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:18.049235 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n_odh-ai-gateway-infra(d6abd316-2c0e-43ec-9c51-9600f833b021)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" Nov 02 07:00:18.050335 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.050314 5025 generic.go:356] "Generic (PLEG): container finished" podID="83ac4602-feb7-4f17-8634-c31db023e71a" containerID="6f56d7866b49197b5456c28408f61e6822b815adb22624261a17387f3eb38244" exitCode=60 Nov 02 07:00:18.050424 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.050386 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" event={"ID":"83ac4602-feb7-4f17-8634-c31db023e71a","Type":"ContainerDied","Data":"6f56d7866b49197b5456c28408f61e6822b815adb22624261a17387f3eb38244"} Nov 02 07:00:18.050755 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.050734 5025 scope.go:122] "RemoveContainer" containerID="6f56d7866b49197b5456c28408f61e6822b815adb22624261a17387f3eb38244" Nov 02 07:00:18.051034 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:18.051014 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6_odh-ai-gateway-infra(83ac4602-feb7-4f17-8634-c31db023e71a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" Nov 02 07:00:18.052104 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.052086 5025 generic.go:356] "Generic (PLEG): container finished" podID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerID="12727229aa1e9828012810cd75eb88696a6d77d3aad9a1380ef53873c4942465" exitCode=60 Nov 02 07:00:18.052168 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.052115 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" event={"ID":"300f0119-015c-45eb-8758-f8c7ee065e7f","Type":"ContainerDied","Data":"12727229aa1e9828012810cd75eb88696a6d77d3aad9a1380ef53873c4942465"} Nov 02 07:00:18.052403 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.052389 5025 scope.go:122] "RemoveContainer" containerID="12727229aa1e9828012810cd75eb88696a6d77d3aad9a1380ef53873c4942465" Nov 02 07:00:18.052620 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:18.052602 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx_odh-ai-gateway-infra(300f0119-015c-45eb-8758-f8c7ee065e7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" Nov 02 07:00:18.062521 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.062500 5025 scope.go:122] "RemoveContainer" containerID="359a77a3abeb31ce7dde6e6590b9c8df460c82df160533f6f2c77c25e6be7b76" Nov 02 07:00:18.074272 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.074243 5025 scope.go:122] "RemoveContainer" containerID="16a9d20f47f4515f4637b87cd0d6aca2423b7813909e274ec324d2ac8ef0b827" Nov 02 07:00:18.292858 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.292830 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" Nov 02 07:00:18.466846 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.466802 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-kube-api-access-mx49t\" (UniqueName: \"kubernetes.io/projected/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-kube-api-access-mx49t\") pod \"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6\" (UID: \"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6\") " Nov 02 07:00:18.467056 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.466960 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-maas-api-service-ca\") pod \"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6\" (UID: \"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6\") " Nov 02 07:00:18.467551 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.467521 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-maas-api-service-ca" pod "31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" (UID: "31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:18.468199 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.468176 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-maas-api-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:18.469264 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.469240 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-kube-api-access-mx49t" pod "31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" (UID: "31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6"). InnerVolumeSpecName "kube-api-access-mx49t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:18.569040 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.568972 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx49t\" (UniqueName: \"kubernetes.io/projected/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6-kube-api-access-mx49t\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:18.620127 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:18.611368 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:19.059298 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.059261 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" Nov 02 07:00:19.059298 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.059291 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64" event={"ID":"31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6","Type":"ContainerDied","Data":"f40d038a3fabaf912eb64cb0b24c3103c8e478fdfdfa3f9946375eca29ddc4fa"} Nov 02 07:00:19.059841 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.059363 5025 scope.go:122] "RemoveContainer" containerID="2722d78a6ab9121a4aedd1559b666dceb00e7d1927242eeaf780f35b41d48574" Nov 02 07:00:19.066436 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.066406 5025 generic.go:356] "Generic (PLEG): container finished" podID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerID="58c68c4b8c515579b6317d568c2d84b93bfb2a5cec9988cd34e4fbc676009ae7" exitCode=60 Nov 02 07:00:19.066550 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.066446 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" event={"ID":"2ddb53c7-b78a-42ff-9e2a-2661872d1890","Type":"ContainerDied","Data":"58c68c4b8c515579b6317d568c2d84b93bfb2a5cec9988cd34e4fbc676009ae7"} Nov 02 07:00:19.066887 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.066870 5025 scope.go:122] "RemoveContainer" containerID="58c68c4b8c515579b6317d568c2d84b93bfb2a5cec9988cd34e4fbc676009ae7" Nov 02 07:00:19.067206 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:19.067185 5025 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25_odh-ai-gateway-infra(2ddb53c7-b78a-42ff-9e2a-2661872d1890)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" Nov 02 07:00:19.070378 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.070359 5025 scope.go:122] "RemoveContainer" containerID="f3b191ce255c207a6c1a89ee15179d6ab8835fb07a8930dd15fbe4d03d9c3299" Nov 02 07:00:19.291684 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.291655 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n"] Nov 02 07:00:19.325994 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.325907 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64"] Nov 02 07:00:19.330119 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.330033 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-6ff85e-29893380-pxp64"] Nov 02 07:00:19.421907 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.421880 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" Nov 02 07:00:19.429758 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.429728 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6"] Nov 02 07:00:19.478243 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.478207 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6abd316-2c0e-43ec-9c51-9600f833b021-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6abd316-2c0e-43ec-9c51-9600f833b021-maas-api-service-ca\") pod \"d6abd316-2c0e-43ec-9c51-9600f833b021\" (UID: \"d6abd316-2c0e-43ec-9c51-9600f833b021\") " Nov 02 07:00:19.478430 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.478362 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6abd316-2c0e-43ec-9c51-9600f833b021-kube-api-access-lbtm7\" (UniqueName: \"kubernetes.io/projected/d6abd316-2c0e-43ec-9c51-9600f833b021-kube-api-access-lbtm7\") pod \"d6abd316-2c0e-43ec-9c51-9600f833b021\" (UID: \"d6abd316-2c0e-43ec-9c51-9600f833b021\") " Nov 02 07:00:19.478637 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.478610 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6abd316-2c0e-43ec-9c51-9600f833b021-maas-api-service-ca" pod "d6abd316-2c0e-43ec-9c51-9600f833b021" (UID: "d6abd316-2c0e-43ec-9c51-9600f833b021"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:19.478884 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.478859 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6abd316-2c0e-43ec-9c51-9600f833b021-maas-api-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:19.480635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.480613 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6abd316-2c0e-43ec-9c51-9600f833b021-kube-api-access-lbtm7" pod "d6abd316-2c0e-43ec-9c51-9600f833b021" (UID: "d6abd316-2c0e-43ec-9c51-9600f833b021"). InnerVolumeSpecName "kube-api-access-lbtm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:19.546873 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.546841 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx"] Nov 02 07:00:19.579606 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.579540 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbtm7\" (UniqueName: \"kubernetes.io/projected/d6abd316-2c0e-43ec-9c51-9600f833b021-kube-api-access-lbtm7\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:19.587295 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.587271 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" Nov 02 07:00:19.678684 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.678645 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" path="/var/lib/kubelet/pods/31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6/volumes" Nov 02 07:00:19.680341 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.680313 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ac4602-feb7-4f17-8634-c31db023e71a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ac4602-feb7-4f17-8634-c31db023e71a-maas-api-service-ca\") pod \"83ac4602-feb7-4f17-8634-c31db023e71a\" (UID: \"83ac4602-feb7-4f17-8634-c31db023e71a\") " Nov 02 07:00:19.680483 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.680371 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ac4602-feb7-4f17-8634-c31db023e71a-kube-api-access-bvfr7\" (UniqueName: \"kubernetes.io/projected/83ac4602-feb7-4f17-8634-c31db023e71a-kube-api-access-bvfr7\") pod \"83ac4602-feb7-4f17-8634-c31db023e71a\" (UID: \"83ac4602-feb7-4f17-8634-c31db023e71a\") " Nov 02 07:00:19.680734 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.680693 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ac4602-feb7-4f17-8634-c31db023e71a-maas-api-service-ca" pod "83ac4602-feb7-4f17-8634-c31db023e71a" (UID: "83ac4602-feb7-4f17-8634-c31db023e71a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:19.680848 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.680828 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83ac4602-feb7-4f17-8634-c31db023e71a-maas-api-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:19.682509 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.682487 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ac4602-feb7-4f17-8634-c31db023e71a-kube-api-access-bvfr7" pod "83ac4602-feb7-4f17-8634-c31db023e71a" (UID: "83ac4602-feb7-4f17-8634-c31db023e71a"). InnerVolumeSpecName "kube-api-access-bvfr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:19.692630 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.692605 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" Nov 02 07:00:19.781159 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.781131 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/300f0119-015c-45eb-8758-f8c7ee065e7f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/300f0119-015c-45eb-8758-f8c7ee065e7f-maas-api-service-ca\") pod \"300f0119-015c-45eb-8758-f8c7ee065e7f\" (UID: \"300f0119-015c-45eb-8758-f8c7ee065e7f\") " Nov 02 07:00:19.781335 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.781271 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/300f0119-015c-45eb-8758-f8c7ee065e7f-kube-api-access-wmpfz\" (UniqueName: \"kubernetes.io/projected/300f0119-015c-45eb-8758-f8c7ee065e7f-kube-api-access-wmpfz\") pod \"300f0119-015c-45eb-8758-f8c7ee065e7f\" (UID: \"300f0119-015c-45eb-8758-f8c7ee065e7f\") " Nov 02 07:00:19.781508 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.781485 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/300f0119-015c-45eb-8758-f8c7ee065e7f-maas-api-service-ca" pod "300f0119-015c-45eb-8758-f8c7ee065e7f" (UID: "300f0119-015c-45eb-8758-f8c7ee065e7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:19.781570 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.781538 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvfr7\" (UniqueName: \"kubernetes.io/projected/83ac4602-feb7-4f17-8634-c31db023e71a-kube-api-access-bvfr7\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:19.783317 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.783299 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/300f0119-015c-45eb-8758-f8c7ee065e7f-kube-api-access-wmpfz" pod "300f0119-015c-45eb-8758-f8c7ee065e7f" (UID: "300f0119-015c-45eb-8758-f8c7ee065e7f"). InnerVolumeSpecName "kube-api-access-wmpfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:19.882923 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.882848 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/300f0119-015c-45eb-8758-f8c7ee065e7f-maas-api-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:19.882923 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:19.882876 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmpfz\" (UniqueName: \"kubernetes.io/projected/300f0119-015c-45eb-8758-f8c7ee065e7f-kube-api-access-wmpfz\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:20.075917 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.075887 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" Nov 02 07:00:20.075917 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.075900 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n" event={"ID":"d6abd316-2c0e-43ec-9c51-9600f833b021","Type":"ContainerDied","Data":"da0c2362eef375b2f5a7c15b88e15460bf11281a3aba97ca2df357837928666a"} Nov 02 07:00:20.076446 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.075947 5025 scope.go:122] "RemoveContainer" containerID="99b308e177eed017b8b6b744abc766e2c12b54409ff7234590b28781b66ca7fe" Nov 02 07:00:20.077333 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.077261 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" Nov 02 07:00:20.077333 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.077269 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6" event={"ID":"83ac4602-feb7-4f17-8634-c31db023e71a","Type":"ContainerDied","Data":"7e58c9bde2457dca118dc17916b28a45c137cc26e21a152490a884c443f95ebb"} Nov 02 07:00:20.079022 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.078906 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" Nov 02 07:00:20.079022 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.078901 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx" event={"ID":"300f0119-015c-45eb-8758-f8c7ee065e7f","Type":"ContainerDied","Data":"87331e5dbc32d3cc9e80f247818a5171460829fb5391b318024007e1ff6db308"} Nov 02 07:00:20.086915 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.086896 5025 scope.go:122] "RemoveContainer" containerID="6f56d7866b49197b5456c28408f61e6822b815adb22624261a17387f3eb38244" Nov 02 07:00:20.095592 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.095566 5025 scope.go:122] "RemoveContainer" containerID="12727229aa1e9828012810cd75eb88696a6d77d3aad9a1380ef53873c4942465" Nov 02 07:00:20.229703 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.229667 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25"] Nov 02 07:00:20.234036 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.234011 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6"] Nov 02 07:00:20.242941 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.242917 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-8bc131-29893380-58kk6"] Nov 02 07:00:20.349002 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.348964 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx"] Nov 02 07:00:20.351186 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.351163 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-6e2657-29893380-cbrrx"] Nov 02 07:00:20.363844 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.363823 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" Nov 02 07:00:20.387232 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.387202 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ddb53c7-b78a-42ff-9e2a-2661872d1890-kube-api-access-rmls2\" (UniqueName: \"kubernetes.io/projected/2ddb53c7-b78a-42ff-9e2a-2661872d1890-kube-api-access-rmls2\") pod \"2ddb53c7-b78a-42ff-9e2a-2661872d1890\" (UID: \"2ddb53c7-b78a-42ff-9e2a-2661872d1890\") " Nov 02 07:00:20.387452 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.387434 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ddb53c7-b78a-42ff-9e2a-2661872d1890-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddb53c7-b78a-42ff-9e2a-2661872d1890-maas-api-service-ca\") pod \"2ddb53c7-b78a-42ff-9e2a-2661872d1890\" (UID: \"2ddb53c7-b78a-42ff-9e2a-2661872d1890\") " Nov 02 07:00:20.387768 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.387737 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ddb53c7-b78a-42ff-9e2a-2661872d1890-maas-api-service-ca" pod "2ddb53c7-b78a-42ff-9e2a-2661872d1890" (UID: "2ddb53c7-b78a-42ff-9e2a-2661872d1890"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:20.389165 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.389144 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ddb53c7-b78a-42ff-9e2a-2661872d1890-kube-api-access-rmls2" pod "2ddb53c7-b78a-42ff-9e2a-2661872d1890" (UID: "2ddb53c7-b78a-42ff-9e2a-2661872d1890"). InnerVolumeSpecName "kube-api-access-rmls2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:20.459473 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.459440 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n"] Nov 02 07:00:20.461229 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.461208 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-5a85b0-29893380-56r9n"] Nov 02 07:00:20.488689 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.488628 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ddb53c7-b78a-42ff-9e2a-2661872d1890-maas-api-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:20.488689 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:20.488652 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmls2\" (UniqueName: \"kubernetes.io/projected/2ddb53c7-b78a-42ff-9e2a-2661872d1890-kube-api-access-rmls2\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:21.089068 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:21.089039 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" event={"ID":"2ddb53c7-b78a-42ff-9e2a-2661872d1890","Type":"ContainerDied","Data":"56f00b37f1f2ad5534cc76d475cd6ad92ac1216715882e14f43e1e1857d3f10c"} Nov 02 07:00:21.089068 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:21.089061 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25" Nov 02 07:00:21.089553 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:21.089124 5025 scope.go:122] "RemoveContainer" containerID="58c68c4b8c515579b6317d568c2d84b93bfb2a5cec9988cd34e4fbc676009ae7" Nov 02 07:00:21.247949 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:21.247915 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25"] Nov 02 07:00:21.251948 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:21.251925 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-581608-29893380-fxq25"] Nov 02 07:00:21.667842 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:21.667807 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" path="/var/lib/kubelet/pods/2ddb53c7-b78a-42ff-9e2a-2661872d1890/volumes" Nov 02 07:00:21.668257 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:21.668244 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" path="/var/lib/kubelet/pods/300f0119-015c-45eb-8758-f8c7ee065e7f/volumes" Nov 02 07:00:21.668608 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:21.668595 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" path="/var/lib/kubelet/pods/83ac4602-feb7-4f17-8634-c31db023e71a/volumes" Nov 02 07:00:21.669247 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:21.669233 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" path="/var/lib/kubelet/pods/d6abd316-2c0e-43ec-9c51-9600f833b021/volumes" Nov 02 07:00:23.196422 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:23.191319 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:27.105432 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:27.105389 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" podUID="49f36379-dc5e-434c-84be-4266642f39ca" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 02 07:00:27.797423 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:27.797369 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:28.182747 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:28.182707 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:29.656060 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:29.656025 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx"] Nov 02 07:00:29.656457 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:29.656246 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" containerID="cri-o://b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da" gracePeriod=30 Nov 02 07:00:29.703430 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:29.703399 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt"] Nov 02 07:00:29.703587 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:29.703574 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" containerID="cri-o://09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2" gracePeriod=30 Nov 02 07:00:32.377253 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:32.377192 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:32.706423 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:32.706388 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:00:33.477492 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:33.477470 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 07:00:33.527925 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:33.527852 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f36379-dc5e-434c-84be-4266642f39ca-kube-api-access-f86sc\" (UniqueName: \"kubernetes.io/projected/49f36379-dc5e-434c-84be-4266642f39ca-kube-api-access-f86sc\") pod \"49f36379-dc5e-434c-84be-4266642f39ca\" (UID: \"49f36379-dc5e-434c-84be-4266642f39ca\") " Nov 02 07:00:33.528056 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:33.527957 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49f36379-dc5e-434c-84be-4266642f39ca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f36379-dc5e-434c-84be-4266642f39ca-plugins-config-volume\") pod \"49f36379-dc5e-434c-84be-4266642f39ca\" (UID: \"49f36379-dc5e-434c-84be-4266642f39ca\") " Nov 02 07:00:33.528320 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:33.528298 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49f36379-dc5e-434c-84be-4266642f39ca-plugins-config-volume" pod "49f36379-dc5e-434c-84be-4266642f39ca" (UID: "49f36379-dc5e-434c-84be-4266642f39ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:33.529898 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:33.529876 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f36379-dc5e-434c-84be-4266642f39ca-kube-api-access-f86sc" pod "49f36379-dc5e-434c-84be-4266642f39ca" (UID: "49f36379-dc5e-434c-84be-4266642f39ca"). InnerVolumeSpecName "kube-api-access-f86sc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:33.629216 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:33.629185 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49f36379-dc5e-434c-84be-4266642f39ca-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:33.629216 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:33.629209 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f86sc\" (UniqueName: \"kubernetes.io/projected/49f36379-dc5e-434c-84be-4266642f39ca-kube-api-access-f86sc\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:34.163220 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:34.163181 5025 generic.go:356] "Generic (PLEG): container finished" podID="49f36379-dc5e-434c-84be-4266642f39ca" containerID="34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b" exitCode=0 Nov 02 07:00:34.163409 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:34.163226 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" Nov 02 07:00:34.163409 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:34.163264 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" event={"ID":"49f36379-dc5e-434c-84be-4266642f39ca","Type":"ContainerDied","Data":"34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b"} Nov 02 07:00:34.163409 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:34.163314 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2" event={"ID":"49f36379-dc5e-434c-84be-4266642f39ca","Type":"ContainerDied","Data":"8bec395897724b04f6e1075596ab5def701b106745eb80accf886c57f63d4819"} Nov 02 07:00:34.163409 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:34.163335 5025 scope.go:122] "RemoveContainer" containerID="34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b" Nov 02 07:00:34.172195 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:34.172176 5025 scope.go:122] "RemoveContainer" containerID="34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b" Nov 02 07:00:34.172464 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:34.172445 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b\": container with ID starting with 34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b not found: ID does not exist" containerID="34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b" Nov 02 07:00:34.172513 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:34.172472 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b"} err="failed to get container status \"34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b\": rpc error: code = NotFound desc = could not find container \"34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b\": container with ID starting with 34180f6f11b689085b7fdb2d246024fa319309751335f0cb4a974b2a26329d0b not found: ID does not exist" Nov 02 07:00:34.320413 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:34.320378 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2"] Nov 02 07:00:34.367869 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:34.367834 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-581608-6659767c8c-dd5l2"] Nov 02 07:00:34.960754 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:34.960728 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 07:00:34.983261 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:34.983238 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 07:00:35.043284 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043249 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-credential-socket\") pod \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " Nov 02 07:00:35.043430 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043324 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istiod-ca-cert\") pod \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " Nov 02 07:00:35.043430 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043357 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-podinfo\") pod \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " Nov 02 07:00:35.043430 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043400 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-podinfo\") pod \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " Nov 02 07:00:35.043618 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043428 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-envoy\") pod \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " Nov 02 07:00:35.043618 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043478 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-token\" (UniqueName: \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-token\") pod \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " Nov 02 07:00:35.043618 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043512 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-token\" (UniqueName: \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-token\") pod \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " Nov 02 07:00:35.043618 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043549 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-data\") pod \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " Nov 02 07:00:35.043618 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043555 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-credential-socket" pod "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" (UID: "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.043618 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043582 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-certs\") pod \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " Nov 02 07:00:35.043618 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043612 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-data\") pod \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " Nov 02 07:00:35.043962 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043651 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-kube-api-access-dnh26\" (UniqueName: \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-kube-api-access-dnh26\") pod \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " Nov 02 07:00:35.043962 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043699 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-socket\") pod \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " Nov 02 07:00:35.043962 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043712 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istiod-ca-cert" pod "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" (UID: "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:35.043962 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043759 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-socket\") pod \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " Nov 02 07:00:35.043962 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043836 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-envoy\") pod \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\" (UID: \"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c\") " Nov 02 07:00:35.043962 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043882 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istiod-ca-cert\") pod \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " Nov 02 07:00:35.043962 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043950 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-certs" pod "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" (UID: "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.044348 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043959 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-certs\") pod \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " Nov 02 07:00:35.044348 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.043976 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-data" pod "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" (UID: "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.044348 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.044086 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-data" pod "3951ca8a-a28d-410d-bfad-a5ea1e49fd51" (UID: "3951ca8a-a28d-410d-bfad-a5ea1e49fd51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.044348 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.044193 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-certs" pod "3951ca8a-a28d-410d-bfad-a5ea1e49fd51" (UID: "3951ca8a-a28d-410d-bfad-a5ea1e49fd51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.044556 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.044360 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-kube-api-access-7w562\" (UniqueName: \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-kube-api-access-7w562\") pod \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " Nov 02 07:00:35.044556 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.044408 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-credential-socket\") pod \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\" (UID: \"3951ca8a-a28d-410d-bfad-a5ea1e49fd51\") " Nov 02 07:00:35.044781 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.044751 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istiod-ca-cert" pod "3951ca8a-a28d-410d-bfad-a5ea1e49fd51" (UID: "3951ca8a-a28d-410d-bfad-a5ea1e49fd51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:35.045015 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.044825 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-socket" pod "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" (UID: "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.045015 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.044940 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-credential-socket" pod "3951ca8a-a28d-410d-bfad-a5ea1e49fd51" (UID: "3951ca8a-a28d-410d-bfad-a5ea1e49fd51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.045015 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.044950 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-socket" pod "3951ca8a-a28d-410d-bfad-a5ea1e49fd51" (UID: "3951ca8a-a28d-410d-bfad-a5ea1e49fd51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.045279 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.045256 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.045338 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.045286 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.045338 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.045304 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.045338 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.045319 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.045338 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.045334 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.045550 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.045352 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.045550 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.045368 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.045550 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.045381 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.045550 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.045395 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.045550 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.045409 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.046294 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.046268 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-envoy" pod "3951ca8a-a28d-410d-bfad-a5ea1e49fd51" (UID: "3951ca8a-a28d-410d-bfad-a5ea1e49fd51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.046403 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.046298 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-token" pod "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" (UID: "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.046463 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.046431 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-token" pod "3951ca8a-a28d-410d-bfad-a5ea1e49fd51" (UID: "3951ca8a-a28d-410d-bfad-a5ea1e49fd51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.046865 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.046848 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-podinfo" pod "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" (UID: "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:00:35.047115 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.047070 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-kube-api-access-dnh26" pod "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" (UID: "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c"). InnerVolumeSpecName "kube-api-access-dnh26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.047454 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.047432 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-kube-api-access-7w562" pod "3951ca8a-a28d-410d-bfad-a5ea1e49fd51" (UID: "3951ca8a-a28d-410d-bfad-a5ea1e49fd51"). InnerVolumeSpecName "kube-api-access-7w562". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:35.048010 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.047984 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-envoy" pod "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" (UID: "ec594c99-c66a-4ca2-b126-db0cbb8e9a1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:35.048114 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.048062 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-podinfo" pod "3951ca8a-a28d-410d-bfad-a5ea1e49fd51" (UID: "3951ca8a-a28d-410d-bfad-a5ea1e49fd51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:00:35.146479 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.146443 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.146479 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.146475 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w562\" (UniqueName: \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-kube-api-access-7w562\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.146679 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.146489 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.146679 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.146502 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.146679 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.146515 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.146679 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.146526 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3951ca8a-a28d-410d-bfad-a5ea1e49fd51-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.146679 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.146536 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.146679 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.146548 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnh26\" (UniqueName: \"kubernetes.io/projected/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c-kube-api-access-dnh26\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:35.169966 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.169924 5025 generic.go:356] "Generic (PLEG): container finished" podID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerID="09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2" exitCode=0 Nov 02 07:00:35.170164 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.169984 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" Nov 02 07:00:35.170164 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.170011 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" event={"ID":"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c","Type":"ContainerDied","Data":"09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2"} Nov 02 07:00:35.170164 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.170106 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt" event={"ID":"ec594c99-c66a-4ca2-b126-db0cbb8e9a1c","Type":"ContainerDied","Data":"f2c9b7c30b6ea9ef88b63f325f91a3a13de03a7eaf1c9ccdff9fcaf04b6e1eec"} Nov 02 07:00:35.170164 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.170144 5025 scope.go:122] "RemoveContainer" containerID="09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2" Nov 02 07:00:35.175231 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.175199 5025 generic.go:356] "Generic (PLEG): container finished" podID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerID="b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da" exitCode=0 Nov 02 07:00:35.175373 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.175252 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" Nov 02 07:00:35.175373 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.175289 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" event={"ID":"3951ca8a-a28d-410d-bfad-a5ea1e49fd51","Type":"ContainerDied","Data":"b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da"} Nov 02 07:00:35.175373 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.175345 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx" event={"ID":"3951ca8a-a28d-410d-bfad-a5ea1e49fd51","Type":"ContainerDied","Data":"6166a0e8b6c4b43f961794e4ccc47090422597abd6fe2f7c150ec07d28fd4d1d"} Nov 02 07:00:35.182940 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.182916 5025 scope.go:122] "RemoveContainer" containerID="09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2" Nov 02 07:00:35.183277 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:35.183256 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2\": container with ID starting with 09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2 not found: ID does not exist" containerID="09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2" Nov 02 07:00:35.183388 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.183283 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2"} err="failed to get container status \"09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2\": rpc error: code = NotFound desc = could not find container \"09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2\": container with ID starting with 09dcf9d833e246dce8ec092ad1b8af4cb7dd67bc4b8c08cea3f48a6e48e693f2 not found: ID does not exist" Nov 02 07:00:35.183388 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.183300 5025 scope.go:122] "RemoveContainer" containerID="b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da" Nov 02 07:00:35.194029 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.194008 5025 scope.go:122] "RemoveContainer" containerID="b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da" Nov 02 07:00:35.194305 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:35.194285 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da\": container with ID starting with b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da not found: ID does not exist" containerID="b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da" Nov 02 07:00:35.194349 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.194314 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da"} err="failed to get container status \"b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da\": rpc error: code = NotFound desc = could not find container \"b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da\": container with ID starting with b90123d0c3430d400eed2e777b8f8788387757ff91a5e17818eacfa3e4e9c8da not found: ID does not exist" Nov 02 07:00:35.348989 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.348956 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt"] Nov 02 07:00:35.356564 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.356539 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nrmmt"] Nov 02 07:00:35.460699 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.460622 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx"] Nov 02 07:00:35.464637 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.464608 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-ct8dx"] Nov 02 07:00:35.668424 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.668392 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" path="/var/lib/kubelet/pods/3951ca8a-a28d-410d-bfad-a5ea1e49fd51/volumes" Nov 02 07:00:35.668849 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.668834 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f36379-dc5e-434c-84be-4266642f39ca" path="/var/lib/kubelet/pods/49f36379-dc5e-434c-84be-4266642f39ca/volumes" Nov 02 07:00:35.669301 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:35.669286 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" path="/var/lib/kubelet/pods/ec594c99-c66a-4ca2-b126-db0cbb8e9a1c/volumes" Nov 02 07:00:37.081482 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:37.081448 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:39.178135 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:39.176381 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:39.628781 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:39.628679 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:39.807216 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:39.807180 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:39.987168 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:39.987070 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:41.355586 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:41.355558 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:42.466411 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:42.466376 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:42.877583 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:42.877509 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:43.093707 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:43.093599 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:43.688343 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:43.688306 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:43.991277 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:43.991168 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:44.317441 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:44.316801 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:44.498104 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:44.498046 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:45.015359 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:45.015326 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:46.093194 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:46.093060 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:46.282662 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:46.282628 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:49.372782 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:49.372740 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:49.609590 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:49.609490 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:49.822089 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:49.822037 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:49.882370 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:49.882338 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8"] Nov 02 07:00:49.882560 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:49.882536 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" containerID="cri-o://1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed" gracePeriod=30 Nov 02 07:00:50.028719 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:50.028681 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:52.264017 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:52.263979 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:52.551486 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:52.551408 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6"] Nov 02 07:00:52.551658 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:52.551637 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" containerID="cri-o://5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b" gracePeriod=30 Nov 02 07:00:52.555520 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:52.555478 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:52.774462 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:52.774424 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:53.783432 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:53.783393 5025 patch_prober.go:29] interesting pod/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 07:00:53.783803 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:53.783462 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 07:00:54.258528 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:54.257949 5025 patch_prober.go:29] interesting pod/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 07:00:54.258528 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:54.258012 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 07:00:54.273667 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:54.273621 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:55.131284 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.131257 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 07:00:55.205096 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205047 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-data\") pod \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " Nov 02 07:00:55.205285 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205105 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-envoy\") pod \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " Nov 02 07:00:55.205285 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205142 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-podinfo\") pod \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " Nov 02 07:00:55.205285 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205165 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-socket\") pod \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " Nov 02 07:00:55.205285 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205201 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-kube-api-access-gthlw\" (UniqueName: \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-kube-api-access-gthlw\") pod \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " Nov 02 07:00:55.205510 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205290 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-credential-socket\") pod \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " Nov 02 07:00:55.205510 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205346 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-data" pod "90a63809-f2cc-4157-80e4-d5cba8ddb07d" (UID: "90a63809-f2cc-4157-80e4-d5cba8ddb07d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:55.205510 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205375 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istiod-ca-cert\") pod \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " Nov 02 07:00:55.205510 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205426 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-certs\") pod \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " Nov 02 07:00:55.205510 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205477 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-token\" (UniqueName: \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-token\") pod \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\" (UID: \"90a63809-f2cc-4157-80e4-d5cba8ddb07d\") " Nov 02 07:00:55.205780 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205521 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-socket" pod "90a63809-f2cc-4157-80e4-d5cba8ddb07d" (UID: "90a63809-f2cc-4157-80e4-d5cba8ddb07d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:55.205780 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205711 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-certs" pod "90a63809-f2cc-4157-80e4-d5cba8ddb07d" (UID: "90a63809-f2cc-4157-80e4-d5cba8ddb07d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:55.205780 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205716 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istiod-ca-cert" pod "90a63809-f2cc-4157-80e4-d5cba8ddb07d" (UID: "90a63809-f2cc-4157-80e4-d5cba8ddb07d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:55.205938 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205894 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-credential-socket" pod "90a63809-f2cc-4157-80e4-d5cba8ddb07d" (UID: "90a63809-f2cc-4157-80e4-d5cba8ddb07d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:55.205999 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205950 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:55.205999 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205969 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:55.205999 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205984 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:55.205999 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.205995 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:55.207516 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.207494 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-kube-api-access-gthlw" pod "90a63809-f2cc-4157-80e4-d5cba8ddb07d" (UID: "90a63809-f2cc-4157-80e4-d5cba8ddb07d"). InnerVolumeSpecName "kube-api-access-gthlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:55.207516 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.207509 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-token" pod "90a63809-f2cc-4157-80e4-d5cba8ddb07d" (UID: "90a63809-f2cc-4157-80e4-d5cba8ddb07d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:55.207711 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.207689 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-envoy" pod "90a63809-f2cc-4157-80e4-d5cba8ddb07d" (UID: "90a63809-f2cc-4157-80e4-d5cba8ddb07d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:55.207765 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.207695 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-podinfo" pod "90a63809-f2cc-4157-80e4-d5cba8ddb07d" (UID: "90a63809-f2cc-4157-80e4-d5cba8ddb07d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:00:55.306146 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.305955 5025 generic.go:356] "Generic (PLEG): container finished" podID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerID="1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed" exitCode=0 Nov 02 07:00:55.306146 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.306020 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" Nov 02 07:00:55.306146 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.306026 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" event={"ID":"90a63809-f2cc-4157-80e4-d5cba8ddb07d","Type":"ContainerDied","Data":"1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed"} Nov 02 07:00:55.306146 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.306101 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8" event={"ID":"90a63809-f2cc-4157-80e4-d5cba8ddb07d","Type":"ContainerDied","Data":"c3b2b7eb7a7e13d65e2b6dad40a35ad57911ca98d0416a3a90e1f2926c79b827"} Nov 02 07:00:55.306146 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.306133 5025 scope.go:122] "RemoveContainer" containerID="1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed" Nov 02 07:00:55.306899 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.306782 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:55.306899 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.306808 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:55.306899 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.306823 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90a63809-f2cc-4157-80e4-d5cba8ddb07d-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:55.306899 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.306839 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gthlw\" (UniqueName: \"kubernetes.io/projected/90a63809-f2cc-4157-80e4-d5cba8ddb07d-kube-api-access-gthlw\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:55.306899 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.306854 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90a63809-f2cc-4157-80e4-d5cba8ddb07d-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:55.317893 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.317854 5025 scope.go:122] "RemoveContainer" containerID="896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe" Nov 02 07:00:55.327796 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.327768 5025 scope.go:122] "RemoveContainer" containerID="1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed" Nov 02 07:00:55.328060 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:55.328041 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed\": container with ID starting with 1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed not found: ID does not exist" containerID="1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed" Nov 02 07:00:55.328180 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.328086 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed"} err="failed to get container status \"1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed\": rpc error: code = NotFound desc = could not find container \"1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed\": container with ID starting with 1f20eafe37f019eaf34229826eaca6d5273d98ddb4bc9eacde10e9c3fcbebaed not found: ID does not exist" Nov 02 07:00:55.328180 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.328110 5025 scope.go:122] "RemoveContainer" containerID="896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe" Nov 02 07:00:55.328347 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:55.328330 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe\": container with ID starting with 896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe not found: ID does not exist" containerID="896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe" Nov 02 07:00:55.328408 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.328355 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe"} err="failed to get container status \"896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe\": rpc error: code = NotFound desc = could not find container \"896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe\": container with ID starting with 896b7fb1e8775d013de2be95fef0b261c51822a277124816f233f7bf9d74e7fe not found: ID does not exist" Nov 02 07:00:55.462838 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.462757 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8"] Nov 02 07:00:55.466812 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.466785 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a2ba11-openshift-default-86879967f5-d5jq8"] Nov 02 07:00:55.667805 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:55.667770 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" path="/var/lib/kubelet/pods/90a63809-f2cc-4157-80e4-d5cba8ddb07d/volumes" Nov 02 07:00:57.807307 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.807283 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 07:00:57.932276 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932243 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-socket\") pod \"36972b30-f157-4a06-aa99-55a5e941643c\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " Nov 02 07:00:57.932432 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932303 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-kube-api-access-7kd7t\" (UniqueName: \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-kube-api-access-7kd7t\") pod \"36972b30-f157-4a06-aa99-55a5e941643c\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " Nov 02 07:00:57.932432 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932333 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-certs\") pod \"36972b30-f157-4a06-aa99-55a5e941643c\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " Nov 02 07:00:57.932432 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932406 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36972b30-f157-4a06-aa99-55a5e941643c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36972b30-f157-4a06-aa99-55a5e941643c-istio-podinfo\") pod \"36972b30-f157-4a06-aa99-55a5e941643c\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " Nov 02 07:00:57.932593 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932441 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-data\") pod \"36972b30-f157-4a06-aa99-55a5e941643c\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " Nov 02 07:00:57.932593 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932469 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36972b30-f157-4a06-aa99-55a5e941643c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36972b30-f157-4a06-aa99-55a5e941643c-istiod-ca-cert\") pod \"36972b30-f157-4a06-aa99-55a5e941643c\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " Nov 02 07:00:57.932593 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932531 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-envoy\") pod \"36972b30-f157-4a06-aa99-55a5e941643c\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " Nov 02 07:00:57.932593 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932570 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-socket" pod "36972b30-f157-4a06-aa99-55a5e941643c" (UID: "36972b30-f157-4a06-aa99-55a5e941643c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:57.932797 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932624 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-istio-token\" (UniqueName: \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-istio-token\") pod \"36972b30-f157-4a06-aa99-55a5e941643c\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " Nov 02 07:00:57.932797 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932656 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-credential-socket\") pod \"36972b30-f157-4a06-aa99-55a5e941643c\" (UID: \"36972b30-f157-4a06-aa99-55a5e941643c\") " Nov 02 07:00:57.932905 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932781 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-data" pod "36972b30-f157-4a06-aa99-55a5e941643c" (UID: "36972b30-f157-4a06-aa99-55a5e941643c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:57.932905 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932801 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-certs" pod "36972b30-f157-4a06-aa99-55a5e941643c" (UID: "36972b30-f157-4a06-aa99-55a5e941643c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:57.932905 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932884 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36972b30-f157-4a06-aa99-55a5e941643c-istiod-ca-cert" pod "36972b30-f157-4a06-aa99-55a5e941643c" (UID: "36972b30-f157-4a06-aa99-55a5e941643c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:00:57.933059 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.932997 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-credential-socket" pod "36972b30-f157-4a06-aa99-55a5e941643c" (UID: "36972b30-f157-4a06-aa99-55a5e941643c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:57.933249 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.933225 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:57.933383 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.933263 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36972b30-f157-4a06-aa99-55a5e941643c-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:57.933383 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.933280 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:57.933383 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.933292 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:57.933383 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.933304 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:57.934564 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.934543 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-kube-api-access-7kd7t" pod "36972b30-f157-4a06-aa99-55a5e941643c" (UID: "36972b30-f157-4a06-aa99-55a5e941643c"). InnerVolumeSpecName "kube-api-access-7kd7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:57.934676 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.934559 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36972b30-f157-4a06-aa99-55a5e941643c-istio-podinfo" pod "36972b30-f157-4a06-aa99-55a5e941643c" (UID: "36972b30-f157-4a06-aa99-55a5e941643c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:00:57.934970 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.934954 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-envoy" pod "36972b30-f157-4a06-aa99-55a5e941643c" (UID: "36972b30-f157-4a06-aa99-55a5e941643c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:00:57.935035 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:57.934959 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-istio-token" pod "36972b30-f157-4a06-aa99-55a5e941643c" (UID: "36972b30-f157-4a06-aa99-55a5e941643c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:00:58.034538 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.034505 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36972b30-f157-4a06-aa99-55a5e941643c-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:58.034538 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.034532 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:58.034538 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.034544 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kd7t\" (UniqueName: \"kubernetes.io/projected/36972b30-f157-4a06-aa99-55a5e941643c-kube-api-access-7kd7t\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:58.034768 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.034555 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36972b30-f157-4a06-aa99-55a5e941643c-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:00:58.324877 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.324784 5025 generic.go:356] "Generic (PLEG): container finished" podID="36972b30-f157-4a06-aa99-55a5e941643c" containerID="5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b" exitCode=0 Nov 02 07:00:58.324877 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.324830 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" Nov 02 07:00:58.324877 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.324866 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" event={"ID":"36972b30-f157-4a06-aa99-55a5e941643c","Type":"ContainerDied","Data":"5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b"} Nov 02 07:00:58.325183 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.324949 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6" event={"ID":"36972b30-f157-4a06-aa99-55a5e941643c","Type":"ContainerDied","Data":"8d7d63c3638483993cedb8419c9ab1a771b7bc547a0ec7906df71123afadbeda"} Nov 02 07:00:58.325183 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.324972 5025 scope.go:122] "RemoveContainer" containerID="5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b" Nov 02 07:00:58.334359 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.334342 5025 scope.go:122] "RemoveContainer" containerID="96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c" Nov 02 07:00:58.342338 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.342322 5025 scope.go:122] "RemoveContainer" containerID="5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b" Nov 02 07:00:58.342584 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:58.342564 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b\": container with ID starting with 5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b not found: ID does not exist" containerID="5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b" Nov 02 07:00:58.342639 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.342593 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b"} err="failed to get container status \"5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b\": rpc error: code = NotFound desc = could not find container \"5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b\": container with ID starting with 5d58d4c474a914b9b4bd86e6899f8abdf02662cbd94c08bb7e88a3e318f23f0b not found: ID does not exist" Nov 02 07:00:58.342639 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.342610 5025 scope.go:122] "RemoveContainer" containerID="96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c" Nov 02 07:00:58.342833 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:00:58.342814 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c\": container with ID starting with 96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c not found: ID does not exist" containerID="96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c" Nov 02 07:00:58.342891 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.342842 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c"} err="failed to get container status \"96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c\": rpc error: code = NotFound desc = could not find container \"96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c\": container with ID starting with 96d8f95cfc11b16d259965b4a9c098c3da9202bcf7c492571c1580a8edd6ef9c not found: ID does not exist" Nov 02 07:00:58.481520 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.481490 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6"] Nov 02 07:00:58.529040 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:58.529012 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-8bc131-openshift-default-7f6fd4bfc9-fm4d6"] Nov 02 07:00:59.211525 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:59.211432 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:00:59.668002 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:00:59.667965 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36972b30-f157-4a06-aa99-55a5e941643c" path="/var/lib/kubelet/pods/36972b30-f157-4a06-aa99-55a5e941643c/volumes" Nov 02 07:01:02.494523 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:02.494483 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:02.693487 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:02.693457 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:02.706185 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:02.706154 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:01:04.288904 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:04.288866 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:04.797929 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:04.793152 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:04.970895 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:04.970813 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:09.255143 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:09.255106 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:14.461585 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:14.461182 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:18.316106 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.316057 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9"] Nov 02 07:01:18.316640 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.316310 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" podUID="2dfb3a3f-ea26-4cad-a870-c5f4c733d480" containerName="maas-api" containerID="cri-o://d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e" gracePeriod=30 Nov 02 07:01:18.462918 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.462883 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:18.582376 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.582314 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 07:01:18.746855 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.746823 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-kube-api-access-4trcw\" (UniqueName: \"kubernetes.io/projected/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-kube-api-access-4trcw\") pod \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " Nov 02 07:01:18.747041 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.746891 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-openshift-service-ca\") pod \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " Nov 02 07:01:18.747041 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.746965 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-maas-api-tls\") pod \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " Nov 02 07:01:18.747041 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.746991 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-metrics-certs\") pod \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\" (UID: \"2dfb3a3f-ea26-4cad-a870-c5f4c733d480\") " Nov 02 07:01:18.747361 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.747327 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-openshift-service-ca" pod "2dfb3a3f-ea26-4cad-a870-c5f4c733d480" (UID: "2dfb3a3f-ea26-4cad-a870-c5f4c733d480"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:18.748989 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.748963 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-metrics-certs" pod "2dfb3a3f-ea26-4cad-a870-c5f4c733d480" (UID: "2dfb3a3f-ea26-4cad-a870-c5f4c733d480"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:18.749071 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.748998 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-maas-api-tls" pod "2dfb3a3f-ea26-4cad-a870-c5f4c733d480" (UID: "2dfb3a3f-ea26-4cad-a870-c5f4c733d480"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:18.749238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.749173 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-kube-api-access-4trcw" pod "2dfb3a3f-ea26-4cad-a870-c5f4c733d480" (UID: "2dfb3a3f-ea26-4cad-a870-c5f4c733d480"). InnerVolumeSpecName "kube-api-access-4trcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:18.848746 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.848650 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4trcw\" (UniqueName: \"kubernetes.io/projected/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-kube-api-access-4trcw\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:18.848746 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.848684 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:18.848746 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.848698 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:18.848746 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:18.848710 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dfb3a3f-ea26-4cad-a870-c5f4c733d480-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:19.428672 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.428634 5025 generic.go:356] "Generic (PLEG): container finished" podID="2dfb3a3f-ea26-4cad-a870-c5f4c733d480" containerID="d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e" exitCode=0 Nov 02 07:01:19.428672 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.428672 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" event={"ID":"2dfb3a3f-ea26-4cad-a870-c5f4c733d480","Type":"ContainerDied","Data":"d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e"} Nov 02 07:01:19.429262 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.428687 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" Nov 02 07:01:19.429262 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.428714 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9" event={"ID":"2dfb3a3f-ea26-4cad-a870-c5f4c733d480","Type":"ContainerDied","Data":"c84912db04f242aef2152dc8eb6522d70e18e6fbe7d203506a9952918bd1db27"} Nov 02 07:01:19.429262 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.428740 5025 scope.go:122] "RemoveContainer" containerID="d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e" Nov 02 07:01:19.437806 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.437788 5025 scope.go:122] "RemoveContainer" containerID="d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e" Nov 02 07:01:19.438053 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:19.438035 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e\": container with ID starting with d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e not found: ID does not exist" containerID="d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e" Nov 02 07:01:19.438127 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.438058 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e"} err="failed to get container status \"d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e\": rpc error: code = NotFound desc = could not find container \"d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e\": container with ID starting with d51f50e29a28940b74f66eed4208df53e20e965a94c4a0b1d764b5336540bf2e not found: ID does not exist" Nov 02 07:01:19.560324 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.560287 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:19.585621 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.585594 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9"] Nov 02 07:01:19.589330 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.589307 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-581608-6996cbdc89-2lxc9"] Nov 02 07:01:19.668643 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.668612 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dfb3a3f-ea26-4cad-a870-c5f4c733d480" path="/var/lib/kubelet/pods/2dfb3a3f-ea26-4cad-a870-c5f4c733d480/volumes" Nov 02 07:01:19.910613 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:19.910506 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:20.181881 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:20.181834 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:20.700550 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:20.700522 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:20.890881 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:20.890849 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:21.075150 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:21.075049 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:21.318706 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:21.318646 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:21.966137 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:21.966106 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:22.253906 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:22.253819 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:22.492310 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:22.492276 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:22.680674 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:22.680644 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:23.177697 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:23.177668 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:26.781523 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:26.781472 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:27.035213 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:27.035132 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:27.219772 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:27.219737 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:27.473023 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:27.472993 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:28.336318 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:28.336287 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:28.579622 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:28.579591 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr"] Nov 02 07:01:28.579860 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:28.579831 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" containerID="cri-o://2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301" gracePeriod=30 Nov 02 07:01:28.651268 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:28.651179 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:28.865734 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:28.865697 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:29.067947 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:29.067906 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:29.974734 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:29.974702 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:30.138560 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:30.138529 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269"] Nov 02 07:01:30.138781 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:30.138759 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" containerID="cri-o://a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a" gracePeriod=30 Nov 02 07:01:30.210656 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:30.210612 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:32.374497 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:32.374437 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:32.706600 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:32.706565 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:01:33.817877 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.817852 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 07:01:33.906153 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906055 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-envoy\") pod \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " Nov 02 07:01:33.906153 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906125 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-credential-socket\") pod \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " Nov 02 07:01:33.906374 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906167 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-certs\") pod \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " Nov 02 07:01:33.906374 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906190 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-token\" (UniqueName: \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-token\") pod \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " Nov 02 07:01:33.906374 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906221 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istiod-ca-cert\") pod \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " Nov 02 07:01:33.906374 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906283 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-podinfo\") pod \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " Nov 02 07:01:33.906374 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906309 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-data\") pod \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " Nov 02 07:01:33.906374 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906325 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-socket\") pod \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " Nov 02 07:01:33.906374 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906347 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-kube-api-access-9rgbp\" (UniqueName: \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-kube-api-access-9rgbp\") pod \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\" (UID: \"d0c0d533-68c3-4b4c-9769-fa8a193cba17\") " Nov 02 07:01:33.906719 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906475 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-credential-socket" pod "d0c0d533-68c3-4b4c-9769-fa8a193cba17" (UID: "d0c0d533-68c3-4b4c-9769-fa8a193cba17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:33.906719 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906491 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-certs" pod "d0c0d533-68c3-4b4c-9769-fa8a193cba17" (UID: "d0c0d533-68c3-4b4c-9769-fa8a193cba17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:33.906719 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906662 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-data" pod "d0c0d533-68c3-4b4c-9769-fa8a193cba17" (UID: "d0c0d533-68c3-4b4c-9769-fa8a193cba17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:33.906719 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906691 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istiod-ca-cert" pod "d0c0d533-68c3-4b4c-9769-fa8a193cba17" (UID: "d0c0d533-68c3-4b4c-9769-fa8a193cba17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:33.906719 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906710 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:33.906981 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906729 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:33.906981 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.906882 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-socket" pod "d0c0d533-68c3-4b4c-9769-fa8a193cba17" (UID: "d0c0d533-68c3-4b4c-9769-fa8a193cba17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:33.908329 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.908303 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-token" pod "d0c0d533-68c3-4b4c-9769-fa8a193cba17" (UID: "d0c0d533-68c3-4b4c-9769-fa8a193cba17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:33.908433 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.908363 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-envoy" pod "d0c0d533-68c3-4b4c-9769-fa8a193cba17" (UID: "d0c0d533-68c3-4b4c-9769-fa8a193cba17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:33.908475 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.908458 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-kube-api-access-9rgbp" pod "d0c0d533-68c3-4b4c-9769-fa8a193cba17" (UID: "d0c0d533-68c3-4b4c-9769-fa8a193cba17"). InnerVolumeSpecName "kube-api-access-9rgbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:33.908650 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.908623 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-podinfo" pod "d0c0d533-68c3-4b4c-9769-fa8a193cba17" (UID: "d0c0d533-68c3-4b4c-9769-fa8a193cba17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:01:33.972511 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:33.972481 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2"] Nov 02 07:01:34.007701 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.007669 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:34.007701 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.007697 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:34.007701 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.007707 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:34.007908 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.007715 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:34.007908 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.007723 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:34.007908 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.007732 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rgbp\" (UniqueName: \"kubernetes.io/projected/d0c0d533-68c3-4b4c-9769-fa8a193cba17-kube-api-access-9rgbp\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:34.007908 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.007741 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0c0d533-68c3-4b4c-9769-fa8a193cba17-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:34.182329 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.182296 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:34.499741 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.499646 5025 generic.go:356] "Generic (PLEG): container finished" podID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerID="2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301" exitCode=0 Nov 02 07:01:34.499741 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.499717 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" Nov 02 07:01:34.499964 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.499737 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" event={"ID":"d0c0d533-68c3-4b4c-9769-fa8a193cba17","Type":"ContainerDied","Data":"2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301"} Nov 02 07:01:34.499964 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.499819 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr" event={"ID":"d0c0d533-68c3-4b4c-9769-fa8a193cba17","Type":"ContainerDied","Data":"c00fbd38cdea6624fc29560988b73e6526719ac7be5272bdd93a29ed7884c204"} Nov 02 07:01:34.499964 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.499841 5025 scope.go:122] "RemoveContainer" containerID="2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301" Nov 02 07:01:34.509520 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.509502 5025 scope.go:122] "RemoveContainer" containerID="18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4" Nov 02 07:01:34.517214 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.517196 5025 scope.go:122] "RemoveContainer" containerID="2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301" Nov 02 07:01:34.517510 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:34.517487 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301\": container with ID starting with 2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301 not found: ID does not exist" containerID="2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301" Nov 02 07:01:34.517578 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.517521 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301"} err="failed to get container status \"2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301\": rpc error: code = NotFound desc = could not find container \"2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301\": container with ID starting with 2631d40b3c61c0f7f5713366581fcbc686f8c0cf977c98d115c291870eae8301 not found: ID does not exist" Nov 02 07:01:34.517578 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.517546 5025 scope.go:122] "RemoveContainer" containerID="18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4" Nov 02 07:01:34.517800 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:34.517783 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4\": container with ID starting with 18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4 not found: ID does not exist" containerID="18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4" Nov 02 07:01:34.517840 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.517806 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4"} err="failed to get container status \"18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4\": rpc error: code = NotFound desc = could not find container \"18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4\": container with ID starting with 18a72dea8b2726eefb8fe6dbca1a0db3f1fd7a8690aeb6a459790944abcf01f4 not found: ID does not exist" Nov 02 07:01:34.655761 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.655724 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr"] Nov 02 07:01:34.703010 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:34.702984 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5a85b0-openshift-default-6cd5c8cb87-p4btr"] Nov 02 07:01:35.385281 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.385256 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 07:01:35.421526 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.421503 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-certs\") pod \"63732c47-cea0-4015-b099-8b7b465ccdff\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " Nov 02 07:01:35.421638 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.421537 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-credential-socket\") pod \"63732c47-cea0-4015-b099-8b7b465ccdff\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " Nov 02 07:01:35.421638 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.421566 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-kube-api-access-6v6zt\" (UniqueName: \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-kube-api-access-6v6zt\") pod \"63732c47-cea0-4015-b099-8b7b465ccdff\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " Nov 02 07:01:35.421755 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.421701 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63732c47-cea0-4015-b099-8b7b465ccdff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63732c47-cea0-4015-b099-8b7b465ccdff-istiod-ca-cert\") pod \"63732c47-cea0-4015-b099-8b7b465ccdff\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " Nov 02 07:01:35.421755 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.421739 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-istio-token\" (UniqueName: \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-istio-token\") pod \"63732c47-cea0-4015-b099-8b7b465ccdff\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " Nov 02 07:01:35.421864 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.421801 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-data\") pod \"63732c47-cea0-4015-b099-8b7b465ccdff\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " Nov 02 07:01:35.421864 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.421806 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-certs" pod "63732c47-cea0-4015-b099-8b7b465ccdff" (UID: "63732c47-cea0-4015-b099-8b7b465ccdff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:35.421864 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.421820 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-credential-socket" pod "63732c47-cea0-4015-b099-8b7b465ccdff" (UID: "63732c47-cea0-4015-b099-8b7b465ccdff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:35.421864 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.421835 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-envoy\") pod \"63732c47-cea0-4015-b099-8b7b465ccdff\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " Nov 02 07:01:35.422061 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.421871 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-socket\") pod \"63732c47-cea0-4015-b099-8b7b465ccdff\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " Nov 02 07:01:35.422061 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.421918 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63732c47-cea0-4015-b099-8b7b465ccdff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63732c47-cea0-4015-b099-8b7b465ccdff-istio-podinfo\") pod \"63732c47-cea0-4015-b099-8b7b465ccdff\" (UID: \"63732c47-cea0-4015-b099-8b7b465ccdff\") " Nov 02 07:01:35.422061 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.422040 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-data" pod "63732c47-cea0-4015-b099-8b7b465ccdff" (UID: "63732c47-cea0-4015-b099-8b7b465ccdff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:35.422274 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.422059 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63732c47-cea0-4015-b099-8b7b465ccdff-istiod-ca-cert" pod "63732c47-cea0-4015-b099-8b7b465ccdff" (UID: "63732c47-cea0-4015-b099-8b7b465ccdff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:35.422274 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.422186 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-socket" pod "63732c47-cea0-4015-b099-8b7b465ccdff" (UID: "63732c47-cea0-4015-b099-8b7b465ccdff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:35.422375 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.422347 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:35.422375 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.422367 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:35.422487 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.422382 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:35.422487 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.422395 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63732c47-cea0-4015-b099-8b7b465ccdff-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:35.422487 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.422409 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:35.423830 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.423800 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-kube-api-access-6v6zt" pod "63732c47-cea0-4015-b099-8b7b465ccdff" (UID: "63732c47-cea0-4015-b099-8b7b465ccdff"). InnerVolumeSpecName "kube-api-access-6v6zt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:35.423958 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.423914 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-istio-token" pod "63732c47-cea0-4015-b099-8b7b465ccdff" (UID: "63732c47-cea0-4015-b099-8b7b465ccdff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:35.423958 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.423951 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-envoy" pod "63732c47-cea0-4015-b099-8b7b465ccdff" (UID: "63732c47-cea0-4015-b099-8b7b465ccdff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:35.424103 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.423988 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63732c47-cea0-4015-b099-8b7b465ccdff-istio-podinfo" pod "63732c47-cea0-4015-b099-8b7b465ccdff" (UID: "63732c47-cea0-4015-b099-8b7b465ccdff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:01:35.508430 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.508398 5025 generic.go:356] "Generic (PLEG): container finished" podID="63732c47-cea0-4015-b099-8b7b465ccdff" containerID="a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a" exitCode=0 Nov 02 07:01:35.508609 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.508446 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" Nov 02 07:01:35.508609 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.508461 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" event={"ID":"63732c47-cea0-4015-b099-8b7b465ccdff","Type":"ContainerDied","Data":"a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a"} Nov 02 07:01:35.508609 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.508501 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269" event={"ID":"63732c47-cea0-4015-b099-8b7b465ccdff","Type":"ContainerDied","Data":"07c1dbdb7b90a29b38975f689ac0e7ea90ff38c73f2276e2805ef43bdb0e6089"} Nov 02 07:01:35.508609 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.508524 5025 scope.go:122] "RemoveContainer" containerID="a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a" Nov 02 07:01:35.519750 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.519729 5025 scope.go:122] "RemoveContainer" containerID="7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2" Nov 02 07:01:35.523206 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.523183 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v6zt\" (UniqueName: \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-kube-api-access-6v6zt\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:35.523206 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.523204 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63732c47-cea0-4015-b099-8b7b465ccdff-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:35.523379 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.523215 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63732c47-cea0-4015-b099-8b7b465ccdff-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:35.523379 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.523225 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63732c47-cea0-4015-b099-8b7b465ccdff-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:35.527801 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.527787 5025 scope.go:122] "RemoveContainer" containerID="a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a" Nov 02 07:01:35.528092 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:35.528061 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a\": container with ID starting with a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a not found: ID does not exist" containerID="a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a" Nov 02 07:01:35.528161 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.528107 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a"} err="failed to get container status \"a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a\": rpc error: code = NotFound desc = could not find container \"a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a\": container with ID starting with a877c2e4bae649cacdc3aee5554109a256c6fab992f321aa7590f825fb13100a not found: ID does not exist" Nov 02 07:01:35.528161 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.528132 5025 scope.go:122] "RemoveContainer" containerID="7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2" Nov 02 07:01:35.528384 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:35.528368 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2\": container with ID starting with 7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2 not found: ID does not exist" containerID="7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2" Nov 02 07:01:35.528421 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.528390 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2"} err="failed to get container status \"7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2\": rpc error: code = NotFound desc = could not find container \"7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2\": container with ID starting with 7b1c1bce380fe818e43786c28dc1fa2b77ff51422e7e9531ef5b8b6e890686a2 not found: ID does not exist" Nov 02 07:01:35.670126 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.670095 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" path="/var/lib/kubelet/pods/d0c0d533-68c3-4b4c-9769-fa8a193cba17/volumes" Nov 02 07:01:35.670384 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.670370 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269"] Nov 02 07:01:35.670429 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:35.670398 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2e9f81-openshift-default-c986c97bc-8g269"] Nov 02 07:01:37.065194 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:37.065158 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:37.667061 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:37.667029 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" path="/var/lib/kubelet/pods/63732c47-cea0-4015-b099-8b7b465ccdff/volumes" Nov 02 07:01:40.949830 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:40.949799 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8"] Nov 02 07:01:41.048441 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:41.048409 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx"] Nov 02 07:01:41.339264 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:41.339187 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:41.514014 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:41.513975 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:41.985776 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:41.985735 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:43.214109 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:43.214066 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp"] Nov 02 07:01:43.314062 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:43.314028 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8"] Nov 02 07:01:43.600285 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:43.600192 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:43.788718 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:43.788683 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:45.549343 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:45.549305 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:45.731361 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:45.731317 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:45.950740 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:45.950698 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" podUID="b9222860-abff-46e2-b001-1de0837f0213" containerName="payload-processing" containerID="cri-o://c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1" gracePeriod=25 Nov 02 07:01:46.049173 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.049103 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" podUID="10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2" containerName="payload-pre-processing" containerID="cri-o://bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76" gracePeriod=25 Nov 02 07:01:46.196648 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.196621 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 07:01:46.305149 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.305123 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 07:01:46.333099 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.333037 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9222860-abff-46e2-b001-1de0837f0213-kube-api-access-7pk2x\" (UniqueName: \"kubernetes.io/projected/b9222860-abff-46e2-b001-1de0837f0213-kube-api-access-7pk2x\") pod \"b9222860-abff-46e2-b001-1de0837f0213\" (UID: \"b9222860-abff-46e2-b001-1de0837f0213\") " Nov 02 07:01:46.333277 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.333108 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9222860-abff-46e2-b001-1de0837f0213-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9222860-abff-46e2-b001-1de0837f0213-plugins-config-volume\") pod \"b9222860-abff-46e2-b001-1de0837f0213\" (UID: \"b9222860-abff-46e2-b001-1de0837f0213\") " Nov 02 07:01:46.333463 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.333441 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9222860-abff-46e2-b001-1de0837f0213-plugins-config-volume" pod "b9222860-abff-46e2-b001-1de0837f0213" (UID: "b9222860-abff-46e2-b001-1de0837f0213"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:46.335058 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.335034 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9222860-abff-46e2-b001-1de0837f0213-kube-api-access-7pk2x" pod "b9222860-abff-46e2-b001-1de0837f0213" (UID: "b9222860-abff-46e2-b001-1de0837f0213"). InnerVolumeSpecName "kube-api-access-7pk2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:46.434388 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.434352 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-plugins-config-volume\") pod \"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2\" (UID: \"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2\") " Nov 02 07:01:46.434559 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.434541 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-kube-api-access-d6lrb\" (UniqueName: \"kubernetes.io/projected/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-kube-api-access-d6lrb\") pod \"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2\" (UID: \"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2\") " Nov 02 07:01:46.434731 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.434710 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-plugins-config-volume" pod "10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2" (UID: "10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:46.434899 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.434879 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pk2x\" (UniqueName: \"kubernetes.io/projected/b9222860-abff-46e2-b001-1de0837f0213-kube-api-access-7pk2x\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:46.434899 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.434897 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:46.435054 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.434905 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9222860-abff-46e2-b001-1de0837f0213-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:46.436504 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.436486 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-kube-api-access-d6lrb" pod "10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2" (UID: "10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2"). InnerVolumeSpecName "kube-api-access-d6lrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:46.486183 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.486068 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:46.535910 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.535882 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6lrb\" (UniqueName: \"kubernetes.io/projected/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2-kube-api-access-d6lrb\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:46.562211 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.562178 5025 generic.go:356] "Generic (PLEG): container finished" podID="10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2" containerID="bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76" exitCode=0 Nov 02 07:01:46.562644 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.562234 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" Nov 02 07:01:46.562644 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.562260 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" event={"ID":"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2","Type":"ContainerDied","Data":"bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76"} Nov 02 07:01:46.562644 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.562327 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx" event={"ID":"10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2","Type":"ContainerDied","Data":"1410653f82723759447c5b3a395be0b0d33f0b84c41c8211e2f091d71082bfa3"} Nov 02 07:01:46.562644 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.562354 5025 scope.go:122] "RemoveContainer" containerID="bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76" Nov 02 07:01:46.564215 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.564181 5025 generic.go:356] "Generic (PLEG): container finished" podID="b9222860-abff-46e2-b001-1de0837f0213" containerID="c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1" exitCode=0 Nov 02 07:01:46.564343 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.564208 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" event={"ID":"b9222860-abff-46e2-b001-1de0837f0213","Type":"ContainerDied","Data":"c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1"} Nov 02 07:01:46.564343 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.564229 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" Nov 02 07:01:46.564343 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.564257 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8" event={"ID":"b9222860-abff-46e2-b001-1de0837f0213","Type":"ContainerDied","Data":"5ee059c99fe3ce8df34e58894894e2d4b116f0b20441e8b6eac5c65b02e96f44"} Nov 02 07:01:46.572258 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.572240 5025 scope.go:122] "RemoveContainer" containerID="bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76" Nov 02 07:01:46.572559 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:46.572535 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76\": container with ID starting with bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76 not found: ID does not exist" containerID="bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76" Nov 02 07:01:46.572675 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.572567 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76"} err="failed to get container status \"bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76\": rpc error: code = NotFound desc = could not find container \"bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76\": container with ID starting with bf60d4294eb246cdc6e62d8a62b85691ec6132126e09c97393c17da781978e76 not found: ID does not exist" Nov 02 07:01:46.572675 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.572594 5025 scope.go:122] "RemoveContainer" containerID="c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1" Nov 02 07:01:46.581298 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.581279 5025 scope.go:122] "RemoveContainer" containerID="c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1" Nov 02 07:01:46.581574 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:46.581552 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1\": container with ID starting with c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1 not found: ID does not exist" containerID="c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1" Nov 02 07:01:46.581649 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.581584 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1"} err="failed to get container status \"c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1\": rpc error: code = NotFound desc = could not find container \"c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1\": container with ID starting with c58ce850038883106b79bd5019dc8dcdc98746d19a58d206f2c2b7ec0016aef1 not found: ID does not exist" Nov 02 07:01:46.721820 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.721787 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx"] Nov 02 07:01:46.725784 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.725749 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-cdfce9-5cff58fb4c-b2hfx"] Nov 02 07:01:46.828914 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.828832 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8"] Nov 02 07:01:46.834864 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:46.834839 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-cdfce9-7f9555c745-mcjs8"] Nov 02 07:01:47.667976 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:47.667942 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2" path="/var/lib/kubelet/pods/10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2/volumes" Nov 02 07:01:47.668383 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:47.668333 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9222860-abff-46e2-b001-1de0837f0213" path="/var/lib/kubelet/pods/b9222860-abff-46e2-b001-1de0837f0213/volumes" Nov 02 07:01:48.215020 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.214984 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" podUID="b6d476df-b880-4cde-93db-1ce99a8ca8f0" containerName="payload-processing" containerID="cri-o://7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3" gracePeriod=25 Nov 02 07:01:48.314329 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.314294 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" podUID="eff748fe-8003-46c3-a21a-e5c7f690e47a" containerName="payload-pre-processing" containerID="cri-o://03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820" gracePeriod=25 Nov 02 07:01:48.463168 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.463142 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 07:01:48.555934 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.555903 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6d476df-b880-4cde-93db-1ce99a8ca8f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d476df-b880-4cde-93db-1ce99a8ca8f0-plugins-config-volume\") pod \"b6d476df-b880-4cde-93db-1ce99a8ca8f0\" (UID: \"b6d476df-b880-4cde-93db-1ce99a8ca8f0\") " Nov 02 07:01:48.556138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.555999 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6d476df-b880-4cde-93db-1ce99a8ca8f0-kube-api-access-28pck\" (UniqueName: \"kubernetes.io/projected/b6d476df-b880-4cde-93db-1ce99a8ca8f0-kube-api-access-28pck\") pod \"b6d476df-b880-4cde-93db-1ce99a8ca8f0\" (UID: \"b6d476df-b880-4cde-93db-1ce99a8ca8f0\") " Nov 02 07:01:48.556367 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.556340 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6d476df-b880-4cde-93db-1ce99a8ca8f0-plugins-config-volume" pod "b6d476df-b880-4cde-93db-1ce99a8ca8f0" (UID: "b6d476df-b880-4cde-93db-1ce99a8ca8f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:48.558295 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.558272 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6d476df-b880-4cde-93db-1ce99a8ca8f0-kube-api-access-28pck" pod "b6d476df-b880-4cde-93db-1ce99a8ca8f0" (UID: "b6d476df-b880-4cde-93db-1ce99a8ca8f0"). InnerVolumeSpecName "kube-api-access-28pck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:48.579170 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.579144 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 07:01:48.579833 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.579808 5025 generic.go:356] "Generic (PLEG): container finished" podID="eff748fe-8003-46c3-a21a-e5c7f690e47a" containerID="03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820" exitCode=0 Nov 02 07:01:48.579938 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.579884 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" event={"ID":"eff748fe-8003-46c3-a21a-e5c7f690e47a","Type":"ContainerDied","Data":"03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820"} Nov 02 07:01:48.580015 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.579988 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" event={"ID":"eff748fe-8003-46c3-a21a-e5c7f690e47a","Type":"ContainerDied","Data":"cd2529740c35005c763c450201dd4207f6f0f477e8becd51b6352020a2917fbb"} Nov 02 07:01:48.580104 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.580020 5025 scope.go:122] "RemoveContainer" containerID="03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820" Nov 02 07:01:48.581550 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.581523 5025 generic.go:356] "Generic (PLEG): container finished" podID="b6d476df-b880-4cde-93db-1ce99a8ca8f0" containerID="7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3" exitCode=0 Nov 02 07:01:48.581658 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.581556 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" event={"ID":"b6d476df-b880-4cde-93db-1ce99a8ca8f0","Type":"ContainerDied","Data":"7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3"} Nov 02 07:01:48.581658 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.581592 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" event={"ID":"b6d476df-b880-4cde-93db-1ce99a8ca8f0","Type":"ContainerDied","Data":"9cebc4456ce61445f6f93f4b7a9f0dc0f92803cf9c028d1bcbadfffc3a2dc017"} Nov 02 07:01:48.581658 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.581594 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp" Nov 02 07:01:48.589058 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.589038 5025 scope.go:122] "RemoveContainer" containerID="03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820" Nov 02 07:01:48.589362 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:48.589343 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820\": container with ID starting with 03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820 not found: ID does not exist" containerID="03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820" Nov 02 07:01:48.589442 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.589369 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820"} err="failed to get container status \"03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820\": rpc error: code = NotFound desc = could not find container \"03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820\": container with ID starting with 03005e405feaf66aa5944f11a7f103a544e101c3b052376b33b81ffff0ad1820 not found: ID does not exist" Nov 02 07:01:48.589442 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.589385 5025 scope.go:122] "RemoveContainer" containerID="7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3" Nov 02 07:01:48.598416 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.598398 5025 scope.go:122] "RemoveContainer" containerID="7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3" Nov 02 07:01:48.598720 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:48.598689 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3\": container with ID starting with 7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3 not found: ID does not exist" containerID="7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3" Nov 02 07:01:48.598720 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.598715 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3"} err="failed to get container status \"7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3\": rpc error: code = NotFound desc = could not find container \"7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3\": container with ID starting with 7a3039d28b32770a544bde9451130023f82991a39e715b3e934f0430a78c10a3 not found: ID does not exist" Nov 02 07:01:48.657571 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.657549 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eff748fe-8003-46c3-a21a-e5c7f690e47a-kube-api-access-56qq8\" (UniqueName: \"kubernetes.io/projected/eff748fe-8003-46c3-a21a-e5c7f690e47a-kube-api-access-56qq8\") pod \"eff748fe-8003-46c3-a21a-e5c7f690e47a\" (UID: \"eff748fe-8003-46c3-a21a-e5c7f690e47a\") " Nov 02 07:01:48.657677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.657657 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eff748fe-8003-46c3-a21a-e5c7f690e47a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff748fe-8003-46c3-a21a-e5c7f690e47a-plugins-config-volume\") pod \"eff748fe-8003-46c3-a21a-e5c7f690e47a\" (UID: \"eff748fe-8003-46c3-a21a-e5c7f690e47a\") " Nov 02 07:01:48.657828 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.657813 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6d476df-b880-4cde-93db-1ce99a8ca8f0-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:48.657900 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.657828 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28pck\" (UniqueName: \"kubernetes.io/projected/b6d476df-b880-4cde-93db-1ce99a8ca8f0-kube-api-access-28pck\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:48.658034 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.658012 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eff748fe-8003-46c3-a21a-e5c7f690e47a-plugins-config-volume" pod "eff748fe-8003-46c3-a21a-e5c7f690e47a" (UID: "eff748fe-8003-46c3-a21a-e5c7f690e47a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:01:48.659465 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.659448 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eff748fe-8003-46c3-a21a-e5c7f690e47a-kube-api-access-56qq8" pod "eff748fe-8003-46c3-a21a-e5c7f690e47a" (UID: "eff748fe-8003-46c3-a21a-e5c7f690e47a"). InnerVolumeSpecName "kube-api-access-56qq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:48.759281 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.759205 5025 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eff748fe-8003-46c3-a21a-e5c7f690e47a-plugins-config-volume\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:48.759281 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.759231 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56qq8\" (UniqueName: \"kubernetes.io/projected/eff748fe-8003-46c3-a21a-e5c7f690e47a-kube-api-access-56qq8\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:48.794918 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.794889 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp"] Nov 02 07:01:48.799404 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.799379 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-3e96be-76d8697457-b6psp"] Nov 02 07:01:48.994538 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:48.994482 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" containerID="cri-o://5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3" gracePeriod=45 Nov 02 07:01:49.225265 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.225242 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 07:01:49.365048 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.364967 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/158b7fb1-559a-4c01-be74-7359c5633cc0-tls-certs\" (UniqueName: \"kubernetes.io/secret/158b7fb1-559a-4c01-be74-7359c5633cc0-tls-certs\") pod \"158b7fb1-559a-4c01-be74-7359c5633cc0\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " Nov 02 07:01:49.365234 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.365132 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-home\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-home\") pod \"158b7fb1-559a-4c01-be74-7359c5633cc0\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " Nov 02 07:01:49.365234 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.365161 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158b7fb1-559a-4c01-be74-7359c5633cc0-kube-api-access-h7v8t\" (UniqueName: \"kubernetes.io/projected/158b7fb1-559a-4c01-be74-7359c5633cc0-kube-api-access-h7v8t\") pod \"158b7fb1-559a-4c01-be74-7359c5633cc0\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " Nov 02 07:01:49.365234 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.365183 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-tmp-dir\") pod \"158b7fb1-559a-4c01-be74-7359c5633cc0\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " Nov 02 07:01:49.365234 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.365206 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-dshm\") pod \"158b7fb1-559a-4c01-be74-7359c5633cc0\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " Nov 02 07:01:49.365454 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.365242 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-model-cache\") pod \"158b7fb1-559a-4c01-be74-7359c5633cc0\" (UID: \"158b7fb1-559a-4c01-be74-7359c5633cc0\") " Nov 02 07:01:49.365454 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.365429 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-home" pod "158b7fb1-559a-4c01-be74-7359c5633cc0" (UID: "158b7fb1-559a-4c01-be74-7359c5633cc0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.365568 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.365476 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-tmp-dir" pod "158b7fb1-559a-4c01-be74-7359c5633cc0" (UID: "158b7fb1-559a-4c01-be74-7359c5633cc0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.365654 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.365635 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-model-cache" pod "158b7fb1-559a-4c01-be74-7359c5633cc0" (UID: "158b7fb1-559a-4c01-be74-7359c5633cc0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.365654 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.365648 5025 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-home\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.365759 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.365664 5025 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-tmp-dir\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.367485 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.367449 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158b7fb1-559a-4c01-be74-7359c5633cc0-kube-api-access-h7v8t" pod "158b7fb1-559a-4c01-be74-7359c5633cc0" (UID: "158b7fb1-559a-4c01-be74-7359c5633cc0"). InnerVolumeSpecName "kube-api-access-h7v8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:01:49.367595 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.367525 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/158b7fb1-559a-4c01-be74-7359c5633cc0-tls-certs" pod "158b7fb1-559a-4c01-be74-7359c5633cc0" (UID: "158b7fb1-559a-4c01-be74-7359c5633cc0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:01:49.367733 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.367707 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-dshm" pod "158b7fb1-559a-4c01-be74-7359c5633cc0" (UID: "158b7fb1-559a-4c01-be74-7359c5633cc0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:01:49.467436 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.467399 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7v8t\" (UniqueName: \"kubernetes.io/projected/158b7fb1-559a-4c01-be74-7359c5633cc0-kube-api-access-h7v8t\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.467436 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.467437 5025 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-dshm\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.467623 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.467452 5025 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/158b7fb1-559a-4c01-be74-7359c5633cc0-model-cache\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.467623 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.467465 5025 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/158b7fb1-559a-4c01-be74-7359c5633cc0-tls-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:01:49.589237 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.589205 5025 generic.go:356] "Generic (PLEG): container finished" podID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerID="5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3" exitCode=0 Nov 02 07:01:49.589431 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.589243 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" event={"ID":"158b7fb1-559a-4c01-be74-7359c5633cc0","Type":"ContainerDied","Data":"5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3"} Nov 02 07:01:49.589431 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.589288 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" event={"ID":"158b7fb1-559a-4c01-be74-7359c5633cc0","Type":"ContainerDied","Data":"6a61f79c00cf92c803d1be89b1278f457062b3cfd7280644c6d02d3e1f8a36c7"} Nov 02 07:01:49.589431 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.589306 5025 scope.go:122] "RemoveContainer" containerID="5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3" Nov 02 07:01:49.589431 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.589262 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2" Nov 02 07:01:49.590703 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.590681 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8" Nov 02 07:01:49.600411 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.600389 5025 scope.go:122] "RemoveContainer" containerID="5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3" Nov 02 07:01:49.600837 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:01:49.600712 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3\": container with ID starting with 5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3 not found: ID does not exist" containerID="5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3" Nov 02 07:01:49.600837 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.600745 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3"} err="failed to get container status \"5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3\": rpc error: code = NotFound desc = could not find container \"5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3\": container with ID starting with 5c6ee5d669cc79feaa1cfb1872682834bab20a18a348e69fbb55035f20d4dbd3 not found: ID does not exist" Nov 02 07:01:49.667663 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.667624 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6d476df-b880-4cde-93db-1ce99a8ca8f0" path="/var/lib/kubelet/pods/b6d476df-b880-4cde-93db-1ce99a8ca8f0/volumes" Nov 02 07:01:49.745360 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.745332 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2"] Nov 02 07:01:49.792047 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.792020 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-cdfce9/distinct-2-w1-cdfce9-kserve-7f47dcc6f-6ggz2"] Nov 02 07:01:49.853309 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.853279 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8"] Nov 02 07:01:49.859247 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:49.859224 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w6-3e96be-6888894ff-l2bd8"] Nov 02 07:01:51.073816 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:51.073784 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:51.667437 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:51.667407 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" path="/var/lib/kubelet/pods/158b7fb1-559a-4c01-be74-7359c5633cc0/volumes" Nov 02 07:01:51.667779 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:51.667766 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eff748fe-8003-46c3-a21a-e5c7f690e47a" path="/var/lib/kubelet/pods/eff748fe-8003-46c3-a21a-e5c7f690e47a/volumes" Nov 02 07:01:55.672902 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:55.666772 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:01:59.691621 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:01:59.691584 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:00.657191 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:00.657161 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf"] Nov 02 07:02:00.657375 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:00.657353 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" podUID="b89b59af-18df-49fe-90c6-3cd91786547c" containerName="maas-api" containerID="cri-o://ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5" gracePeriod=30 Nov 02 07:02:00.814831 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:00.814792 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:00.922047 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:00.922026 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 07:02:01.084169 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.084136 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b89b59af-18df-49fe-90c6-3cd91786547c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89b59af-18df-49fe-90c6-3cd91786547c-openshift-service-ca\") pod \"b89b59af-18df-49fe-90c6-3cd91786547c\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " Nov 02 07:02:01.084335 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.084240 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b89b59af-18df-49fe-90c6-3cd91786547c-kube-api-access-nkdfs\" (UniqueName: \"kubernetes.io/projected/b89b59af-18df-49fe-90c6-3cd91786547c-kube-api-access-nkdfs\") pod \"b89b59af-18df-49fe-90c6-3cd91786547c\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " Nov 02 07:02:01.084335 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.084280 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-maas-api-tls\") pod \"b89b59af-18df-49fe-90c6-3cd91786547c\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " Nov 02 07:02:01.084335 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.084303 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-metrics-certs\") pod \"b89b59af-18df-49fe-90c6-3cd91786547c\" (UID: \"b89b59af-18df-49fe-90c6-3cd91786547c\") " Nov 02 07:02:01.084590 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.084555 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b89b59af-18df-49fe-90c6-3cd91786547c-openshift-service-ca" pod "b89b59af-18df-49fe-90c6-3cd91786547c" (UID: "b89b59af-18df-49fe-90c6-3cd91786547c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:02:01.084709 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.084688 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b89b59af-18df-49fe-90c6-3cd91786547c-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:01.086489 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.086458 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b89b59af-18df-49fe-90c6-3cd91786547c-kube-api-access-nkdfs" pod "b89b59af-18df-49fe-90c6-3cd91786547c" (UID: "b89b59af-18df-49fe-90c6-3cd91786547c"). InnerVolumeSpecName "kube-api-access-nkdfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:01.086489 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.086468 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-maas-api-tls" pod "b89b59af-18df-49fe-90c6-3cd91786547c" (UID: "b89b59af-18df-49fe-90c6-3cd91786547c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:01.086489 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.086480 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-metrics-certs" pod "b89b59af-18df-49fe-90c6-3cd91786547c" (UID: "b89b59af-18df-49fe-90c6-3cd91786547c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:01.120546 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.120522 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:01.185921 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.185887 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkdfs\" (UniqueName: \"kubernetes.io/projected/b89b59af-18df-49fe-90c6-3cd91786547c-kube-api-access-nkdfs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:01.185921 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.185916 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:01.185921 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.185927 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b89b59af-18df-49fe-90c6-3cd91786547c-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:01.383975 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.383944 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:01.650545 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.650461 5025 generic.go:356] "Generic (PLEG): container finished" podID="b89b59af-18df-49fe-90c6-3cd91786547c" containerID="ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5" exitCode=0 Nov 02 07:02:01.650545 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.650505 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" Nov 02 07:02:01.650736 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.650544 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" event={"ID":"b89b59af-18df-49fe-90c6-3cd91786547c","Type":"ContainerDied","Data":"ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5"} Nov 02 07:02:01.650736 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.650598 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf" event={"ID":"b89b59af-18df-49fe-90c6-3cd91786547c","Type":"ContainerDied","Data":"b4b5d792d2bb793327cde3f9966a09ca88e719b8afb4bc17226234a99552a985"} Nov 02 07:02:01.650736 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.650624 5025 scope.go:122] "RemoveContainer" containerID="ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5" Nov 02 07:02:01.660352 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.660140 5025 scope.go:122] "RemoveContainer" containerID="ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5" Nov 02 07:02:01.660433 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:02:01.660362 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5\": container with ID starting with ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5 not found: ID does not exist" containerID="ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5" Nov 02 07:02:01.660433 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.660385 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5"} err="failed to get container status \"ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5\": rpc error: code = NotFound desc = could not find container \"ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5\": container with ID starting with ca561e2ccfe1f835a723ed9507ca06c753826a3458972a5d242c107d9bb72bf5 not found: ID does not exist" Nov 02 07:02:01.809063 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.809030 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf"] Nov 02 07:02:01.856142 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:01.856114 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-cdfce9-7b77cfc658-5jxlf"] Nov 02 07:02:02.016382 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:02.016350 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:02.302638 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:02.302558 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:02.673881 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:02.673840 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:02.705338 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:02:02.705301 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:02:02.872321 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:02.872252 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:03.203846 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:03.203815 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:03.667389 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:03.667362 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b89b59af-18df-49fe-90c6-3cd91786547c" path="/var/lib/kubelet/pods/b89b59af-18df-49fe-90c6-3cd91786547c/volumes" Nov 02 07:02:07.479273 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:07.479216 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:08.003718 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:08.003681 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:08.206057 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:08.206021 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:08.235771 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:08.235738 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv"] Nov 02 07:02:08.235946 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:08.235928 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" containerID="cri-o://fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff" gracePeriod=30 Nov 02 07:02:08.472680 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:08.472649 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:09.839762 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:09.837961 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:10.023389 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:10.023360 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:10.212860 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:10.212827 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:12.411930 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:12.411888 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:12.428825 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:12.428791 5025 patch_prober.go:29] interesting pod/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 07:02:12.428988 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:12.428856 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 07:02:13.480302 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.480277 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 07:02:13.600460 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600362 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abe717d-5020-42af-9ad7-ab8c166f6936-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abe717d-5020-42af-9ad7-ab8c166f6936-istiod-ca-cert\") pod \"9abe717d-5020-42af-9ad7-ab8c166f6936\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " Nov 02 07:02:13.600460 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600415 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-token\" (UniqueName: \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-token\") pod \"9abe717d-5020-42af-9ad7-ab8c166f6936\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " Nov 02 07:02:13.600715 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600482 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-envoy\") pod \"9abe717d-5020-42af-9ad7-ab8c166f6936\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " Nov 02 07:02:13.600715 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600521 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-data\") pod \"9abe717d-5020-42af-9ad7-ab8c166f6936\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " Nov 02 07:02:13.600715 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600567 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-kube-api-access-z2zzb\" (UniqueName: \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-kube-api-access-z2zzb\") pod \"9abe717d-5020-42af-9ad7-ab8c166f6936\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " Nov 02 07:02:13.600715 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600651 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-podinfo\") pod \"9abe717d-5020-42af-9ad7-ab8c166f6936\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " Nov 02 07:02:13.600715 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600698 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-credential-socket\") pod \"9abe717d-5020-42af-9ad7-ab8c166f6936\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " Nov 02 07:02:13.600984 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600765 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abe717d-5020-42af-9ad7-ab8c166f6936-istiod-ca-cert" pod "9abe717d-5020-42af-9ad7-ab8c166f6936" (UID: "9abe717d-5020-42af-9ad7-ab8c166f6936"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:02:13.600984 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600853 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-data" pod "9abe717d-5020-42af-9ad7-ab8c166f6936" (UID: "9abe717d-5020-42af-9ad7-ab8c166f6936"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:13.600984 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600772 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-socket\") pod \"9abe717d-5020-42af-9ad7-ab8c166f6936\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " Nov 02 07:02:13.600984 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600955 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-certs\") pod \"9abe717d-5020-42af-9ad7-ab8c166f6936\" (UID: \"9abe717d-5020-42af-9ad7-ab8c166f6936\") " Nov 02 07:02:13.601243 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.600999 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-socket" pod "9abe717d-5020-42af-9ad7-ab8c166f6936" (UID: "9abe717d-5020-42af-9ad7-ab8c166f6936"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:13.601243 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.601164 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-credential-socket" pod "9abe717d-5020-42af-9ad7-ab8c166f6936" (UID: "9abe717d-5020-42af-9ad7-ab8c166f6936"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:13.601364 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.601252 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-certs" pod "9abe717d-5020-42af-9ad7-ab8c166f6936" (UID: "9abe717d-5020-42af-9ad7-ab8c166f6936"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:13.601425 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.601412 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abe717d-5020-42af-9ad7-ab8c166f6936-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:13.601485 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.601433 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:13.601485 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.601448 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:13.601485 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.601473 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:13.601645 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.601488 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:13.602929 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.602903 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-token" pod "9abe717d-5020-42af-9ad7-ab8c166f6936" (UID: "9abe717d-5020-42af-9ad7-ab8c166f6936"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:13.602929 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.602918 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-envoy" pod "9abe717d-5020-42af-9ad7-ab8c166f6936" (UID: "9abe717d-5020-42af-9ad7-ab8c166f6936"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:13.603108 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.602994 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-podinfo" pod "9abe717d-5020-42af-9ad7-ab8c166f6936" (UID: "9abe717d-5020-42af-9ad7-ab8c166f6936"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:02:13.603155 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.603124 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-kube-api-access-z2zzb" pod "9abe717d-5020-42af-9ad7-ab8c166f6936" (UID: "9abe717d-5020-42af-9ad7-ab8c166f6936"). InnerVolumeSpecName "kube-api-access-z2zzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:13.702480 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.702448 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:13.702480 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.702477 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2zzb\" (UniqueName: \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-kube-api-access-z2zzb\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:13.702660 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.702492 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:13.702660 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.702504 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abe717d-5020-42af-9ad7-ab8c166f6936-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:13.705591 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.705565 5025 generic.go:356] "Generic (PLEG): container finished" podID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerID="fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff" exitCode=0 Nov 02 07:02:13.705712 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.705619 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" event={"ID":"9abe717d-5020-42af-9ad7-ab8c166f6936","Type":"ContainerDied","Data":"fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff"} Nov 02 07:02:13.705712 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.705654 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" event={"ID":"9abe717d-5020-42af-9ad7-ab8c166f6936","Type":"ContainerDied","Data":"3997780c9f8f3ba1965de263ea65429369415a2b8e822afad50f29a4f8eba360"} Nov 02 07:02:13.705712 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.705675 5025 scope.go:122] "RemoveContainer" containerID="fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff" Nov 02 07:02:13.705712 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.705621 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv" Nov 02 07:02:13.714347 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.714319 5025 scope.go:122] "RemoveContainer" containerID="5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218" Nov 02 07:02:13.721641 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.721621 5025 scope.go:122] "RemoveContainer" containerID="fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff" Nov 02 07:02:13.721912 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:02:13.721894 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff\": container with ID starting with fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff not found: ID does not exist" containerID="fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff" Nov 02 07:02:13.721984 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.721919 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff"} err="failed to get container status \"fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff\": rpc error: code = NotFound desc = could not find container \"fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff\": container with ID starting with fee454a3c5eb13082052768c18f46776d0a109b2b5b03cb11d1a94dc6f532fff not found: ID does not exist" Nov 02 07:02:13.721984 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.721935 5025 scope.go:122] "RemoveContainer" containerID="5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218" Nov 02 07:02:13.722180 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:02:13.722160 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218\": container with ID starting with 5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218 not found: ID does not exist" containerID="5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218" Nov 02 07:02:13.722219 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.722187 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218"} err="failed to get container status \"5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218\": rpc error: code = NotFound desc = could not find container \"5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218\": container with ID starting with 5cdf90cfd8dc44477fea96c1f36c36f0497e027a360ca8601ba530f75ae81218 not found: ID does not exist" Nov 02 07:02:13.822698 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.822666 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv"] Nov 02 07:02:13.823963 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:13.823935 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-3e96be-openshift-default-9b66599f5-7vmdv"] Nov 02 07:02:15.667030 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:15.666997 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" path="/var/lib/kubelet/pods/9abe717d-5020-42af-9ad7-ab8c166f6936/volumes" Nov 02 07:02:17.095530 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:17.095494 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:17.287861 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:17.287820 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:19.249184 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:19.249149 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:20.869660 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:20.869626 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:21.059013 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:21.058983 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:21.237383 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:21.237323 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:22.822164 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:22.822128 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:24.659476 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:24.659441 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:24.913347 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:24.913315 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:25.323293 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:25.323207 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:25.529988 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:25.529953 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:27.559050 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:27.559019 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:28.838171 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:28.838137 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:31.447104 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:31.447053 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:31.670303 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:31.670276 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:31.757162 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:31.757038 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75"] Nov 02 07:02:31.757323 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:31.757264 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" containerID="cri-o://96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416" gracePeriod=30 Nov 02 07:02:31.873752 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:31.873711 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:32.171473 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:32.171439 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:32.369911 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:32.369872 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:32.707834 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:02:32.707801 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:02:37.000560 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.000534 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 07:02:37.136119 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136015 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-kube-api-access-6fcs2\" (UniqueName: \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-kube-api-access-6fcs2\") pod \"288d008c-009e-4543-ae54-4ef60cc18c1a\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " Nov 02 07:02:37.136119 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136059 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-data\") pod \"288d008c-009e-4543-ae54-4ef60cc18c1a\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " Nov 02 07:02:37.136119 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136115 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-certs\") pod \"288d008c-009e-4543-ae54-4ef60cc18c1a\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " Nov 02 07:02:37.136423 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136262 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-envoy\") pod \"288d008c-009e-4543-ae54-4ef60cc18c1a\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " Nov 02 07:02:37.136423 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136312 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-podinfo\") pod \"288d008c-009e-4543-ae54-4ef60cc18c1a\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " Nov 02 07:02:37.136423 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136383 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-socket\") pod \"288d008c-009e-4543-ae54-4ef60cc18c1a\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " Nov 02 07:02:37.136423 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136397 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-certs" pod "288d008c-009e-4543-ae54-4ef60cc18c1a" (UID: "288d008c-009e-4543-ae54-4ef60cc18c1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:37.136423 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136411 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-credential-socket\") pod \"288d008c-009e-4543-ae54-4ef60cc18c1a\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " Nov 02 07:02:37.136423 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136410 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-data" pod "288d008c-009e-4543-ae54-4ef60cc18c1a" (UID: "288d008c-009e-4543-ae54-4ef60cc18c1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:37.136780 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136471 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-token\" (UniqueName: \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-token\") pod \"288d008c-009e-4543-ae54-4ef60cc18c1a\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " Nov 02 07:02:37.136780 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136501 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/288d008c-009e-4543-ae54-4ef60cc18c1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288d008c-009e-4543-ae54-4ef60cc18c1a-istiod-ca-cert\") pod \"288d008c-009e-4543-ae54-4ef60cc18c1a\" (UID: \"288d008c-009e-4543-ae54-4ef60cc18c1a\") " Nov 02 07:02:37.136780 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136629 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-socket" pod "288d008c-009e-4543-ae54-4ef60cc18c1a" (UID: "288d008c-009e-4543-ae54-4ef60cc18c1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:37.136780 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136730 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-credential-socket" pod "288d008c-009e-4543-ae54-4ef60cc18c1a" (UID: "288d008c-009e-4543-ae54-4ef60cc18c1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:37.136992 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136938 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:37.136992 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136942 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/288d008c-009e-4543-ae54-4ef60cc18c1a-istiod-ca-cert" pod "288d008c-009e-4543-ae54-4ef60cc18c1a" (UID: "288d008c-009e-4543-ae54-4ef60cc18c1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:02:37.136992 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136961 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:37.136992 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.136989 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:37.137184 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.137004 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:37.138387 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.138365 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-envoy" pod "288d008c-009e-4543-ae54-4ef60cc18c1a" (UID: "288d008c-009e-4543-ae54-4ef60cc18c1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:02:37.138567 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.138547 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-token" pod "288d008c-009e-4543-ae54-4ef60cc18c1a" (UID: "288d008c-009e-4543-ae54-4ef60cc18c1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:37.138670 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.138653 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-kube-api-access-6fcs2" pod "288d008c-009e-4543-ae54-4ef60cc18c1a" (UID: "288d008c-009e-4543-ae54-4ef60cc18c1a"). InnerVolumeSpecName "kube-api-access-6fcs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:37.138718 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.138659 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-podinfo" pod "288d008c-009e-4543-ae54-4ef60cc18c1a" (UID: "288d008c-009e-4543-ae54-4ef60cc18c1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:02:37.173128 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.173062 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:37.238755 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.238726 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fcs2\" (UniqueName: \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-kube-api-access-6fcs2\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:37.238755 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.238752 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:37.238950 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.238763 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:37.238950 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.238782 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/288d008c-009e-4543-ae54-4ef60cc18c1a-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:37.238950 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.238791 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/288d008c-009e-4543-ae54-4ef60cc18c1a-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:37.814070 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.814034 5025 generic.go:356] "Generic (PLEG): container finished" podID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerID="96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416" exitCode=0 Nov 02 07:02:37.814260 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.814119 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" Nov 02 07:02:37.814260 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.814117 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" event={"ID":"288d008c-009e-4543-ae54-4ef60cc18c1a","Type":"ContainerDied","Data":"96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416"} Nov 02 07:02:37.814260 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.814177 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75" event={"ID":"288d008c-009e-4543-ae54-4ef60cc18c1a","Type":"ContainerDied","Data":"4b832ccd126454d1841433b2e70e7b30fb838549e2e635a888a5ecda7826aaed"} Nov 02 07:02:37.814260 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.814201 5025 scope.go:122] "RemoveContainer" containerID="96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416" Nov 02 07:02:37.822861 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.822845 5025 scope.go:122] "RemoveContainer" containerID="944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c" Nov 02 07:02:37.831715 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.831693 5025 scope.go:122] "RemoveContainer" containerID="96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416" Nov 02 07:02:37.832017 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:02:37.831997 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416\": container with ID starting with 96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416 not found: ID does not exist" containerID="96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416" Nov 02 07:02:37.832164 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.832030 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416"} err="failed to get container status \"96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416\": rpc error: code = NotFound desc = could not find container \"96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416\": container with ID starting with 96f9610a8d991132537c98a440b177c1af4ab17adc6e0511142e08be140a8416 not found: ID does not exist" Nov 02 07:02:37.832164 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.832055 5025 scope.go:122] "RemoveContainer" containerID="944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c" Nov 02 07:02:37.832352 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:02:37.832329 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c\": container with ID starting with 944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c not found: ID does not exist" containerID="944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c" Nov 02 07:02:37.832393 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:37.832361 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c"} err="failed to get container status \"944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c\": rpc error: code = NotFound desc = could not find container \"944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c\": container with ID starting with 944cc15f0b2015e2c6ae4b9c8563e1d1d12c018a92bc6ee5e77e512c8aee958c not found: ID does not exist" Nov 02 07:02:38.018297 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:38.018214 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75"] Nov 02 07:02:38.022340 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:38.022311 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-6e2657-openshift-default-655c476df5-xmn75"] Nov 02 07:02:39.658559 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:39.658522 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:39.666836 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:39.666808 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" path="/var/lib/kubelet/pods/288d008c-009e-4543-ae54-4ef60cc18c1a/volumes" Nov 02 07:02:41.743946 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:41.743911 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:44.544560 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:44.544525 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:44.734238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:44.734201 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:02:45.825829 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.825787 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97"] Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826146 5025 cpu_manager.go:414] "containerMap: removing container" podUID="2dfb3a3f-ea26-4cad-a870-c5f4c733d480" containerName="maas-api" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826158 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dfb3a3f-ea26-4cad-a870-c5f4c733d480" containerName="maas-api" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826170 5025 cpu_manager.go:414] "containerMap: removing container" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826175 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826183 5025 cpu_manager.go:414] "containerMap: removing container" podUID="10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2" containerName="payload-pre-processing" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826188 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2" containerName="payload-pre-processing" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826197 5025 cpu_manager.go:414] "containerMap: removing container" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerName="cleanup" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826202 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerName="cleanup" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826209 5025 cpu_manager.go:414] "containerMap: removing container" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826213 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826219 5025 cpu_manager.go:414] "containerMap: removing container" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826224 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826231 5025 cpu_manager.go:414] "containerMap: removing container" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerName="cleanup" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826235 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerName="cleanup" Nov 02 07:02:45.826238 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826243 5025 cpu_manager.go:414] "containerMap: removing container" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826247 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826255 5025 cpu_manager.go:414] "containerMap: removing container" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826259 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826265 5025 cpu_manager.go:414] "containerMap: removing container" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826269 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826277 5025 cpu_manager.go:414] "containerMap: removing container" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826281 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826287 5025 cpu_manager.go:414] "containerMap: removing container" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826291 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826297 5025 cpu_manager.go:414] "containerMap: removing container" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826300 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826306 5025 cpu_manager.go:414] "containerMap: removing container" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826310 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826317 5025 cpu_manager.go:414] "containerMap: removing container" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826321 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826327 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826331 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826336 5025 cpu_manager.go:414] "containerMap: removing container" podUID="eff748fe-8003-46c3-a21a-e5c7f690e47a" containerName="payload-pre-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826340 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eff748fe-8003-46c3-a21a-e5c7f690e47a" containerName="payload-pre-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826347 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826351 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826357 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b6d476df-b880-4cde-93db-1ce99a8ca8f0" containerName="payload-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826361 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6d476df-b880-4cde-93db-1ce99a8ca8f0" containerName="payload-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826369 5025 cpu_manager.go:414] "containerMap: removing container" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826373 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826380 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826384 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826391 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826395 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826402 5025 cpu_manager.go:414] "containerMap: removing container" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826407 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826412 5025 cpu_manager.go:414] "containerMap: removing container" podUID="89055878-ba75-4384-8290-b3855b20d6af" containerName="main" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826417 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89055878-ba75-4384-8290-b3855b20d6af" containerName="main" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826426 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b9222860-abff-46e2-b001-1de0837f0213" containerName="payload-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826430 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9222860-abff-46e2-b001-1de0837f0213" containerName="payload-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826435 5025 cpu_manager.go:414] "containerMap: removing container" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826439 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826449 5025 cpu_manager.go:414] "containerMap: removing container" podUID="49f36379-dc5e-434c-84be-4266642f39ca" containerName="payload-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826453 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f36379-dc5e-434c-84be-4266642f39ca" containerName="payload-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826459 5025 cpu_manager.go:414] "containerMap: removing container" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826464 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826470 5025 cpu_manager.go:414] "containerMap: removing container" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826474 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826479 5025 cpu_manager.go:414] "containerMap: removing container" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826483 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826491 5025 cpu_manager.go:414] "containerMap: removing container" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826495 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826501 5025 cpu_manager.go:414] "containerMap: removing container" podUID="b89b59af-18df-49fe-90c6-3cd91786547c" containerName="maas-api" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826506 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b89b59af-18df-49fe-90c6-3cd91786547c" containerName="maas-api" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826552 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826558 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="89055878-ba75-4384-8290-b3855b20d6af" containerName="main" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826564 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826569 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9222860-abff-46e2-b001-1de0837f0213" containerName="payload-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826576 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826581 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826586 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f36379-dc5e-434c-84be-4266642f39ca" containerName="payload-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826592 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826596 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826600 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3951ca8a-a28d-410d-bfad-a5ea1e49fd51" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826605 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="63732c47-cea0-4015-b099-8b7b465ccdff" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826611 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826619 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826624 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826629 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b89b59af-18df-49fe-90c6-3cd91786547c" containerName="maas-api" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826636 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dfb3a3f-ea26-4cad-a870-c5f4c733d480" containerName="maas-api" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826640 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826645 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b1f206-c4f2-40bb-a1a7-ce97a7cfd1d2" containerName="payload-pre-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826650 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826656 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826661 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826666 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826671 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826676 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826681 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826685 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826690 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="36972b30-f157-4a06-aa99-55a5e941643c" containerName="istio-proxy" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826694 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826698 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="eff748fe-8003-46c3-a21a-e5c7f690e47a" containerName="payload-pre-processing" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826704 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerName="cleanup" Nov 02 07:02:45.826673 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826708 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" Nov 02 07:02:45.829431 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826713 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6d476df-b880-4cde-93db-1ce99a8ca8f0" containerName="payload-processing" Nov 02 07:02:45.829431 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826718 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="158b7fb1-559a-4c01-be74-7359c5633cc0" containerName="main" Nov 02 07:02:45.829431 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826724 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6abd316-2c0e-43ec-9c51-9600f833b021" containerName="cleanup" Nov 02 07:02:45.829431 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.826729 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec594c99-c66a-4ca2-b126-db0cbb8e9a1c" containerName="istio-proxy" Nov 02 07:02:45.831402 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.831376 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:45.884714 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.884674 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97"] Nov 02 07:02:45.919857 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.919824 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed10bfb-bqk97\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:45.920007 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.919886 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvb5\" (UniqueName: \"kubernetes.io/projected/3d2b131c-f395-43d1-9332-9847676b73a8-kube-api-access-5cvb5\") pod \"maas-api-revoke-sub-7ed10bfb-bqk97\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:45.920007 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.919982 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed10bfb-bqk97\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:45.929715 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.929686 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 02 07:02:45.929855 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:45.929823 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-68l4r\"" Nov 02 07:02:46.021027 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:46.020997 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvb5\" (UniqueName: \"kubernetes.io/projected/3d2b131c-f395-43d1-9332-9847676b73a8-kube-api-access-5cvb5\") pod \"maas-api-revoke-sub-7ed10bfb-bqk97\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:46.021235 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:46.021040 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed10bfb-bqk97\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:46.021235 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:46.021111 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed10bfb-bqk97\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:46.021859 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:46.021833 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ed10bfb-bqk97\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:46.023629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:46.023602 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ed10bfb-bqk97\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:46.076485 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:46.076424 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvb5\" (UniqueName: \"kubernetes.io/projected/3d2b131c-f395-43d1-9332-9847676b73a8-kube-api-access-5cvb5\") pod \"maas-api-revoke-sub-7ed10bfb-bqk97\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:46.144534 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:46.144507 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:46.522867 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:46.522831 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97"] Nov 02 07:02:46.529303 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:02:46.529278 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2b131c_f395_43d1_9332_9847676b73a8.slice/crio-ff8cbb705d066e0f890aafe13529c8e56667e8f49b66869ac0be1233bcb61a7a WatchSource:0}: Error finding container ff8cbb705d066e0f890aafe13529c8e56667e8f49b66869ac0be1233bcb61a7a: Status 404 returned error can't find the container with id ff8cbb705d066e0f890aafe13529c8e56667e8f49b66869ac0be1233bcb61a7a Nov 02 07:02:46.855926 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:46.855834 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" event={"ID":"3d2b131c-f395-43d1-9332-9847676b73a8","Type":"ContainerStarted","Data":"55d8f71aa14397e65798cfe5aba01846117d07d39d992f08b7d25fcc7e76783c"} Nov 02 07:02:46.855926 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:46.855891 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" event={"ID":"3d2b131c-f395-43d1-9332-9847676b73a8","Type":"ContainerStarted","Data":"ff8cbb705d066e0f890aafe13529c8e56667e8f49b66869ac0be1233bcb61a7a"} Nov 02 07:02:46.965084 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:46.965019 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" podStartSLOduration=1.9650069060000002 podStartE2EDuration="1.965006906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:02:46.963963985 +0000 UTC m=+2743.412340759" watchObservedRunningTime="2026-11-02 07:02:46.965006906 +0000 UTC m=+2743.413383639" Nov 02 07:02:47.862453 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:47.862418 5025 generic.go:356] "Generic (PLEG): container finished" podID="3d2b131c-f395-43d1-9332-9847676b73a8" containerID="55d8f71aa14397e65798cfe5aba01846117d07d39d992f08b7d25fcc7e76783c" exitCode=0 Nov 02 07:02:47.862794 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:47.862453 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" event={"ID":"3d2b131c-f395-43d1-9332-9847676b73a8","Type":"ContainerDied","Data":"55d8f71aa14397e65798cfe5aba01846117d07d39d992f08b7d25fcc7e76783c"} Nov 02 07:02:48.999465 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:48.999439 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:49.056259 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.056225 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-openshift-ca\") pod \"3d2b131c-f395-43d1-9332-9847676b73a8\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " Nov 02 07:02:49.056418 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.056313 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-xks-ca\") pod \"3d2b131c-f395-43d1-9332-9847676b73a8\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " Nov 02 07:02:49.056418 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.056336 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2b131c-f395-43d1-9332-9847676b73a8-kube-api-access-5cvb5\" (UniqueName: \"kubernetes.io/projected/3d2b131c-f395-43d1-9332-9847676b73a8-kube-api-access-5cvb5\") pod \"3d2b131c-f395-43d1-9332-9847676b73a8\" (UID: \"3d2b131c-f395-43d1-9332-9847676b73a8\") " Nov 02 07:02:49.056579 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.056551 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-openshift-ca" pod "3d2b131c-f395-43d1-9332-9847676b73a8" (UID: "3d2b131c-f395-43d1-9332-9847676b73a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:02:49.058358 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.058328 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-xks-ca" pod "3d2b131c-f395-43d1-9332-9847676b73a8" (UID: "3d2b131c-f395-43d1-9332-9847676b73a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:02:49.058477 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.058361 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2b131c-f395-43d1-9332-9847676b73a8-kube-api-access-5cvb5" pod "3d2b131c-f395-43d1-9332-9847676b73a8" (UID: "3d2b131c-f395-43d1-9332-9847676b73a8"). InnerVolumeSpecName "kube-api-access-5cvb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:02:49.158002 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.157933 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-openshift-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:49.158002 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.157959 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2b131c-f395-43d1-9332-9847676b73a8-maas-api-xks-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:49.158002 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.157969 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cvb5\" (UniqueName: \"kubernetes.io/projected/3d2b131c-f395-43d1-9332-9847676b73a8-kube-api-access-5cvb5\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:02:49.877799 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.877761 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" event={"ID":"3d2b131c-f395-43d1-9332-9847676b73a8","Type":"ContainerDied","Data":"ff8cbb705d066e0f890aafe13529c8e56667e8f49b66869ac0be1233bcb61a7a"} Nov 02 07:02:49.877799 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.877797 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97" Nov 02 07:02:49.878033 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:02:49.877826 5025 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff8cbb705d066e0f890aafe13529c8e56667e8f49b66869ac0be1233bcb61a7a" Nov 02 07:03:01.253521 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:01.253470 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:02.684314 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:02.684283 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:02.706766 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:03:02.706730 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:03:02.953225 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:02.953193 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:03.572433 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:03.572398 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:09.975368 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:09.975333 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:10.173427 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:10.173394 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:10.358964 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:10.358882 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:13.697312 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:13.697276 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:13.879601 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:13.879568 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:14.652381 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:14.652350 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:20.421350 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:20.421318 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:29.155935 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:29.155883 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:30.991681 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.991651 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm"] Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992006 5025 cpu_manager.go:414] "containerMap: removing container" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerName="cleanup" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992016 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ddb53c7-b78a-42ff-9e2a-2661872d1890" containerName="cleanup" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992026 5025 cpu_manager.go:414] "containerMap: removing container" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992031 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a63809-f2cc-4157-80e4-d5cba8ddb07d" containerName="istio-proxy" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992038 5025 cpu_manager.go:414] "containerMap: removing container" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992043 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="288d008c-009e-4543-ae54-4ef60cc18c1a" containerName="istio-proxy" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992054 5025 cpu_manager.go:414] "containerMap: removing container" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerName="cleanup" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992058 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31aa8fb2-6c55-40a2-80c5-7b12b10e1ae6" containerName="cleanup" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992065 5025 cpu_manager.go:414] "containerMap: removing container" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerName="cleanup" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992070 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerName="cleanup" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992089 5025 cpu_manager.go:414] "containerMap: removing container" podUID="3d2b131c-f395-43d1-9332-9847676b73a8" containerName="revoke-keys" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992094 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2b131c-f395-43d1-9332-9847676b73a8" containerName="revoke-keys" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992103 5025 cpu_manager.go:414] "containerMap: removing container" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" Nov 02 07:03:30.992100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992108 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" Nov 02 07:03:30.992635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992114 5025 cpu_manager.go:414] "containerMap: removing container" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" Nov 02 07:03:30.992635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992118 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" Nov 02 07:03:30.992635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992124 5025 cpu_manager.go:414] "containerMap: removing container" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" containerName="cleanup" Nov 02 07:03:30.992635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992128 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" containerName="cleanup" Nov 02 07:03:30.992635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992186 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="300f0119-015c-45eb-8758-f8c7ee065e7f" containerName="cleanup" Nov 02 07:03:30.992635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992195 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2b131c-f395-43d1-9332-9847676b73a8" containerName="revoke-keys" Nov 02 07:03:30.992635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992204 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0c0d533-68c3-4b4c-9769-fa8a193cba17" containerName="istio-proxy" Nov 02 07:03:30.992635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992211 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abe717d-5020-42af-9ad7-ab8c166f6936" containerName="istio-proxy" Nov 02 07:03:30.992635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992218 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67b2cd6-2ab3-4f88-a576-691f8837773e" containerName="istio-proxy" Nov 02 07:03:30.992635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.992223 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ac4602-feb7-4f17-8634-c31db023e71a" containerName="cleanup" Nov 02 07:03:30.995132 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:30.995116 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.039135 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.039105 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9"] Nov 02 07:03:31.042190 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.042175 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.123168 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.123133 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9"] Nov 02 07:03:31.144175 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144145 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.144305 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144181 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9fm8\" (UniqueName: \"kubernetes.io/projected/24a7b3c3-fb09-4b70-aa73-858f52476799-kube-api-access-n9fm8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.144305 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144208 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.144382 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144305 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.144382 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144355 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.144382 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144376 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.144481 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144464 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.144517 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144490 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.144517 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144512 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.144609 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144535 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjjpp\" (UniqueName: \"kubernetes.io/projected/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-kube-api-access-tjjpp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.144648 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144614 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.144648 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144632 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.144715 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144649 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a7b3c3-fb09-4b70-aa73-858f52476799-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.144715 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144677 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.144780 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144740 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.144780 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144773 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.144848 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144795 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.144848 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.144835 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.183251 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.183225 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm"] Nov 02 07:03:31.246348 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246259 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.246348 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246301 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.246348 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246322 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.246348 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246345 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjjpp\" (UniqueName: \"kubernetes.io/projected/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-kube-api-access-tjjpp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.246677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246372 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.246677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246386 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.246677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246402 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a7b3c3-fb09-4b70-aa73-858f52476799-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.246677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246440 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.246677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246464 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.246677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246494 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.246677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246527 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.246677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246551 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.246677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246591 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.246677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246620 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9fm8\" (UniqueName: \"kubernetes.io/projected/24a7b3c3-fb09-4b70-aa73-858f52476799-kube-api-access-n9fm8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.246677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246657 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.247284 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246689 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.247284 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246732 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.247284 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246740 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.247284 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246759 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.247284 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246867 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.247284 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.246886 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.247284 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.247113 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.248045 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.247908 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.248180 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.248052 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.248342 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.248297 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.248544 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.248513 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.248710 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.248678 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.248804 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.248624 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24a7b3c3-fb09-4b70-aa73-858f52476799-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.249302 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.249279 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.255239 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.255211 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.255535 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.255512 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.257272 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.257248 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.300760 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.300733 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjjpp\" (UniqueName: \"kubernetes.io/projected/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-kube-api-access-tjjpp\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.300895 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.300780 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24a7b3c3-fb09-4b70-aa73-858f52476799-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.301356 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.301340 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c57c5a3-8442-4aa2-967c-a9a80aa58d7b-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-8xsqm\" (UID: \"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.301674 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.301652 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9fm8\" (UniqueName: \"kubernetes.io/projected/24a7b3c3-fb09-4b70-aa73-858f52476799-kube-api-access-n9fm8\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-fmss9\" (UID: \"24a7b3c3-fb09-4b70-aa73-858f52476799\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.307351 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.307332 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:31.358737 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.358645 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:31.725070 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.725041 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm"] Nov 02 07:03:31.726476 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:03:31.726441 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c57c5a3_8442_4aa2_967c_a9a80aa58d7b.slice/crio-921c5b81dfc439782e3818d4ed5f160f6e1aa10804f3073dc0729417ab7a163a WatchSource:0}: Error finding container 921c5b81dfc439782e3818d4ed5f160f6e1aa10804f3073dc0729417ab7a163a: Status 404 returned error can't find the container with id 921c5b81dfc439782e3818d4ed5f160f6e1aa10804f3073dc0729417ab7a163a Nov 02 07:03:31.728531 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.728494 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:03:31.728665 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.728633 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:03:31.728733 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.728681 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:03:31.772378 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.772349 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9"] Nov 02 07:03:31.773701 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:03:31.773676 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a7b3c3_fb09_4b70_aa73_858f52476799.slice/crio-f8632148c99e65040bc0f2b63b4b8746a6dfef6f76c5376b65477efb88ba2044 WatchSource:0}: Error finding container f8632148c99e65040bc0f2b63b4b8746a6dfef6f76c5376b65477efb88ba2044: Status 404 returned error can't find the container with id f8632148c99e65040bc0f2b63b4b8746a6dfef6f76c5376b65477efb88ba2044 Nov 02 07:03:31.775830 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.775799 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:03:31.775907 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.775862 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:03:31.775907 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:31.775890 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:03:32.059446 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.059350 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" event={"ID":"24a7b3c3-fb09-4b70-aa73-858f52476799","Type":"ContainerStarted","Data":"1131185773dd328cf23248f5fd2c8c33fa0f52d6493b32678a1d8d75eb0392e3"} Nov 02 07:03:32.059446 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.059412 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" event={"ID":"24a7b3c3-fb09-4b70-aa73-858f52476799","Type":"ContainerStarted","Data":"f8632148c99e65040bc0f2b63b4b8746a6dfef6f76c5376b65477efb88ba2044"} Nov 02 07:03:32.061137 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.061098 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" event={"ID":"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b","Type":"ContainerStarted","Data":"39d9f53211c46e70faa39d06409ec2171c49ac7efaaecf707e63989bf5db9281"} Nov 02 07:03:32.061248 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.061148 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" event={"ID":"2c57c5a3-8442-4aa2-967c-a9a80aa58d7b","Type":"ContainerStarted","Data":"921c5b81dfc439782e3818d4ed5f160f6e1aa10804f3073dc0729417ab7a163a"} Nov 02 07:03:32.172760 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.172697 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" podStartSLOduration=2.172676878 podStartE2EDuration="2.172676878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:03:32.172636491 +0000 UTC m=+2788.621013267" watchObservedRunningTime="2026-11-02 07:03:32.172676878 +0000 UTC m=+2788.621053635" Nov 02 07:03:32.287530 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.286774 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" podStartSLOduration=2.286753053 podStartE2EDuration="2.286753053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:03:32.284815323 +0000 UTC m=+2788.733192080" watchObservedRunningTime="2026-11-02 07:03:32.286753053 +0000 UTC m=+2788.735129808" Nov 02 07:03:32.313545 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.313447 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:32.316241 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.316207 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 07:03:32.316390 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.316305 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" podUID="2c57c5a3-8442-4aa2-967c-a9a80aa58d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 07:03:32.359416 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.359380 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:32.360947 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.360916 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fmss9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 07:03:32.361071 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:32.360975 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" podUID="24a7b3c3-fb09-4b70-aa73-858f52476799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 07:03:32.748878 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:03:32.748840 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:03:33.309011 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:33.308529 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 07:03:33.309011 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:33.308619 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" podUID="2c57c5a3-8442-4aa2-967c-a9a80aa58d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 07:03:33.359807 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:33.359769 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fmss9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 07:03:33.359979 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:33.359828 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" podUID="24a7b3c3-fb09-4b70-aa73-858f52476799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 07:03:34.324097 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:34.323549 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 07:03:34.324097 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:34.323613 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" podUID="2c57c5a3-8442-4aa2-967c-a9a80aa58d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 07:03:34.359137 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:34.359105 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fmss9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 07:03:34.359299 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:34.359159 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" podUID="24a7b3c3-fb09-4b70-aa73-858f52476799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 07:03:35.316541 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:35.316500 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 07:03:35.316875 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:35.316838 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" podUID="2c57c5a3-8442-4aa2-967c-a9a80aa58d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 07:03:35.359015 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:35.358981 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fmss9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 07:03:35.359411 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:35.359032 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" podUID="24a7b3c3-fb09-4b70-aa73-858f52476799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 07:03:36.312606 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:36.312534 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 07:03:36.312891 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:36.312864 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" podUID="2c57c5a3-8442-4aa2-967c-a9a80aa58d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 07:03:36.359133 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:36.359062 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fmss9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 07:03:36.359530 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:36.359150 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" podUID="24a7b3c3-fb09-4b70-aa73-858f52476799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 07:03:37.311475 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:37.311436 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 07:03:37.311764 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:37.311733 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" podUID="2c57c5a3-8442-4aa2-967c-a9a80aa58d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 07:03:37.359455 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:37.359420 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fmss9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 07:03:37.359835 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:37.359476 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" podUID="24a7b3c3-fb09-4b70-aa73-858f52476799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 07:03:38.308166 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:38.308125 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 07:03:38.308344 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:38.308190 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" podUID="2c57c5a3-8442-4aa2-967c-a9a80aa58d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 07:03:38.359668 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:38.359632 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fmss9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body= Nov 02 07:03:38.360044 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:38.359687 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" podUID="24a7b3c3-fb09-4b70-aa73-858f52476799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" Nov 02 07:03:39.308806 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:39.308763 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" start-of-body= Nov 02 07:03:39.309001 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:39.308833 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" podUID="2c57c5a3-8442-4aa2-967c-a9a80aa58d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": dial tcp 10.133.0.145:15021: connect: connection refused" Nov 02 07:03:40.022946 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:40.022915 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:40.024609 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:40.023663 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:40.024609 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:40.024528 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" Nov 02 07:03:40.311797 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:40.311727 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:40.312038 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:40.312014 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:40.312615 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:40.312580 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" Nov 02 07:03:46.115574 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.115305 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs"] Nov 02 07:03:46.120618 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.120570 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.139859 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.139826 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.140412 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.140389 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqh89\" (UniqueName: \"kubernetes.io/projected/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-kube-api-access-xqh89\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.140677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.140658 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.140851 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.140834 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.141019 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.141001 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.141221 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.141203 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.141410 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.141393 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.141578 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.141559 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.141773 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.141754 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.231933 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.231900 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs"] Nov 02 07:03:46.242782 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.242747 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqh89\" (UniqueName: \"kubernetes.io/projected/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-kube-api-access-xqh89\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.243047 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.243026 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.245291 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.245268 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.246034 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.246016 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.246687 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.246669 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.247308 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.247102 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-workload-certs\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.247526 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.245945 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-workload-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.247751 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.247733 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.248267 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.248187 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-credential-socket\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.249210 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.249137 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.249653 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.249633 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-envoy\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.249876 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.249856 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.250271 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.250252 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.251218 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.251190 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.251521 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.250781 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-data\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.254018 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.253992 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.301587 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.301522 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-istio-token\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.309097 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.309018 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqh89\" (UniqueName: \"kubernetes.io/projected/9b4f70ef-1722-4b66-9a44-5c771b7a24b9-kube-api-access-xqh89\") pod \"maas-default-gateway-openshift-default-55f85cf5d7-nsgcs\" (UID: \"9b4f70ef-1722-4b66-9a44-5c771b7a24b9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.441489 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.441442 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:46.905335 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.904951 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs"] Nov 02 07:03:46.914746 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.913526 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:03:46.914746 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.913616 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:03:46.914746 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:46.913661 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:03:47.141542 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:47.141502 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" event={"ID":"9b4f70ef-1722-4b66-9a44-5c771b7a24b9","Type":"ContainerStarted","Data":"f0f56df6afd3bb8d2faf0a169b7a2871f1c65beb82eb37a585495d155a15b95e"} Nov 02 07:03:47.142161 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:47.142135 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" event={"ID":"9b4f70ef-1722-4b66-9a44-5c771b7a24b9","Type":"ContainerStarted","Data":"1c81db825cbae86b1537852f65156dcc4c6657a138f4521be4e6fec8c8fc7205"} Nov 02 07:03:47.259305 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:47.258192 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" podStartSLOduration=1.258169935 podStartE2EDuration="1.258169935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:03:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:03:47.256764519 +0000 UTC m=+2803.705141286" watchObservedRunningTime="2026-11-02 07:03:47.258169935 +0000 UTC m=+2803.706546691" Nov 02 07:03:47.442417 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:47.442382 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:47.443934 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:47.443902 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 07:03:47.444182 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:47.444156 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" podUID="9b4f70ef-1722-4b66-9a44-5c771b7a24b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 07:03:48.442189 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:48.442154 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 07:03:48.442560 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:48.442211 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" podUID="9b4f70ef-1722-4b66-9a44-5c771b7a24b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 07:03:49.442266 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:49.442224 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 07:03:49.442667 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:49.442281 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" podUID="9b4f70ef-1722-4b66-9a44-5c771b7a24b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 07:03:50.442526 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:50.442490 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 07:03:50.442919 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:50.442545 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" podUID="9b4f70ef-1722-4b66-9a44-5c771b7a24b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 07:03:51.442668 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:51.442623 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 07:03:51.443201 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:51.442686 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" podUID="9b4f70ef-1722-4b66-9a44-5c771b7a24b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 07:03:52.442786 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:52.442752 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 07:03:52.443297 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:52.442811 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" podUID="9b4f70ef-1722-4b66-9a44-5c771b7a24b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 07:03:53.099014 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:53.098976 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:03:53.442810 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:53.442772 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body= Nov 02 07:03:53.443359 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:53.442844 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" podUID="9b4f70ef-1722-4b66-9a44-5c771b7a24b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" Nov 02 07:03:55.200130 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:55.200090 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:55.200572 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:55.200483 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:55.201373 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:55.201345 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" Nov 02 07:03:55.498974 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:03:55.498871 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:04:02.710535 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:04:02.710497 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:04:29.499408 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:04:29.499375 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:04:32.709050 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:04:32.709012 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:04:39.380857 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:04:39.380822 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:05:02.710794 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:05:02.710748 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:05:08.060102 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:05:08.055330 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:05:08.060102 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:05:08.056066 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:05:15.281507 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:05:15.281425 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:05:15.464282 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:05:15.464248 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:05:29.819150 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:05:29.819111 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:05:32.706718 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:05:32.706682 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:06:02.706894 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:06:02.706855 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:06:28.371150 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:06:28.371113 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:06:28.574945 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:06:28.574912 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:06:30.172606 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:06:30.172572 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:06:32.706057 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:06:32.706017 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:06:38.056099 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:06:38.055207 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:06:38.056099 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:06:38.055303 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:07:02.705857 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:07:02.705822 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:07:09.394380 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:09.394348 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:07:11.905655 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:11.905586 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:07:30.549870 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:30.549835 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:07:32.706109 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:07:32.706058 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:07:41.070242 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:41.070199 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fmss9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:07:41.070907 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:41.070872 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" podUID="24a7b3c3-fb09-4b70-aa73-858f52476799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:07:41.256101 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:41.255203 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:07:41.256101 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:41.255295 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" podUID="9b4f70ef-1722-4b66-9a44-5c771b7a24b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:07:41.356156 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:41.355177 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:07:41.356156 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:41.355245 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" podUID="2c57c5a3-8442-4aa2-967c-a9a80aa58d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:07:49.048041 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:49.048003 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97"] Nov 02 07:07:49.095295 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:49.095259 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ed10bfb-bqk97"] Nov 02 07:07:49.667062 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:49.667030 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2b131c-f395-43d1-9332-9847676b73a8" path="/var/lib/kubelet/pods/3d2b131c-f395-43d1-9332-9847676b73a8/volumes" Nov 02 07:07:58.892113 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:07:58.892066 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:08:02.705744 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:08:02.705705 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:08:30.773942 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:08:30.773854 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:08:32.706280 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:08:32.706245 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:09:02.705914 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:09:02.705878 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:09:04.668968 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:09:04.668933 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:09:04.810232 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:09:04.810205 5025 scope.go:122] "RemoveContainer" containerID="55d8f71aa14397e65798cfe5aba01846117d07d39d992f08b7d25fcc7e76783c" Nov 02 07:09:31.217815 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:09:31.217770 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:09:32.780672 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:09:32.780632 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:10:02.706617 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:10:02.706578 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:10:31.499538 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:31.499500 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:10:32.705432 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:10:32.705397 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:10:39.567985 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:39.567651 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:10:40.018926 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.018892 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p"] Nov 02 07:10:40.022427 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.022406 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.056534 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.056496 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-socket\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.056842 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.056819 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-envoy\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.057177 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.057154 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm558\" (UniqueName: \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-kube-api-access-bm558\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.057260 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.057214 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istiod-ca-cert\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.057260 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.057244 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-data\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.057411 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.057389 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-credential-socket\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.057486 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.057430 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-token\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.057568 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.057519 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-certs\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.057624 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.057598 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-podinfo\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.074752 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.074714 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-614951-openshift-default-dockercfg-4ctch\"" Nov 02 07:10:40.124069 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.124036 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p"] Nov 02 07:10:40.156863 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.156795 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:10:40.160678 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.160631 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm558\" (UniqueName: \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-kube-api-access-bm558\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.161367 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.160771 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istiod-ca-cert\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.161367 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.160821 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-data\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.162055 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.162014 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-credential-socket\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.162055 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.162088 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-token\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.162352 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.162148 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-certs\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.162352 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.162178 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-podinfo\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.162352 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.162231 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-socket\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.162352 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.162283 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-envoy\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.162667 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.162486 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istiod-ca-cert\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.162667 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.162521 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-data\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.163005 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.162974 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-socket\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.163134 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.163050 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-credential-socket\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.163134 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.163071 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-certs\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.166117 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.166053 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-podinfo\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.173390 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.173361 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-envoy\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.217912 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.217880 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-token\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.218061 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.217884 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm558\" (UniqueName: \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-kube-api-access-bm558\") pod \"e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.335725 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.335637 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:40.533701 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.533667 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:10:40.748923 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.748893 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p"] Nov 02 07:10:40.750225 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:10:40.750199 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeea2d1f7_93b6_4238_a3d2_8921d0d4784a.slice/crio-44342e069e27f79445f2ec22379eae86c5a43366ad891de9fa8ac0c35f53a595 WatchSource:0}: Error finding container 44342e069e27f79445f2ec22379eae86c5a43366ad891de9fa8ac0c35f53a595: Status 404 returned error can't find the container with id 44342e069e27f79445f2ec22379eae86c5a43366ad891de9fa8ac0c35f53a595 Nov 02 07:10:40.752279 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.752247 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:10:40.752359 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.752309 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:10:40.752359 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:40.752338 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:10:41.355957 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:41.355920 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" event={"ID":"eea2d1f7-93b6-4238-a3d2-8921d0d4784a","Type":"ContainerStarted","Data":"9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131"} Nov 02 07:10:41.356145 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:41.355978 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" event={"ID":"eea2d1f7-93b6-4238-a3d2-8921d0d4784a","Type":"ContainerStarted","Data":"44342e069e27f79445f2ec22379eae86c5a43366ad891de9fa8ac0c35f53a595"} Nov 02 07:10:41.469304 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:41.469247 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" podStartSLOduration=2.469231355 podStartE2EDuration="2.469231355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:10:41.467869777 +0000 UTC m=+3217.916246547" watchObservedRunningTime="2026-11-02 07:10:41.469231355 +0000 UTC m=+3217.917608120" Nov 02 07:10:42.336847 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:42.336812 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:42.341317 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:42.341289 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:42.360753 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:42.360727 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:42.361711 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:42.361694 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:10:44.345909 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:44.345861 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:10:44.570771 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:44.570724 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:10:47.462911 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:47.462858 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:10:47.674246 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:47.667376 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:10:50.275684 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:50.275650 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:10:50.477707 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:50.477671 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:10:54.884129 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:54.881551 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:10:59.418707 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:10:59.418672 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:02.731987 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:11:02.731950 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:11:03.817099 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:03.815653 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:07.468650 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:07.468615 5025 generic.go:356] "Generic (PLEG): container finished" podID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerID="9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131" exitCode=0 Nov 02 07:11:07.469102 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:07.468685 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" event={"ID":"eea2d1f7-93b6-4238-a3d2-8921d0d4784a","Type":"ContainerDied","Data":"9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131"} Nov 02 07:11:07.469102 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:07.469069 5025 scope.go:122] "RemoveContainer" containerID="9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131" Nov 02 07:11:07.470278 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:07.470249 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:11:07.470340 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:07.470326 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:11:07.470375 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:07.470360 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:11:08.475432 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:08.475393 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" event={"ID":"eea2d1f7-93b6-4238-a3d2-8921d0d4784a","Type":"ContainerStarted","Data":"5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838"} Nov 02 07:11:08.536246 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:08.536210 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:09.226548 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:09.226145 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:09.336521 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:09.336477 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:11:09.337761 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:09.337736 5025 patch_prober.go:29] interesting pod/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 07:11:09.337818 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:09.337785 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 07:11:09.806020 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:09.805982 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:10.005972 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:10.005938 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:10.336149 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:10.336111 5025 patch_prober.go:29] interesting pod/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 07:11:10.336331 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:10.336169 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 07:11:11.336434 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:11.336394 5025 patch_prober.go:29] interesting pod/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 07:11:11.336929 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:11.336474 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 07:11:12.336840 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:12.336802 5025 patch_prober.go:29] interesting pod/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Nov 02 07:11:12.337288 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:12.336861 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Nov 02 07:11:12.361506 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:12.361479 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:11:13.017628 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:13.017551 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:13.251093 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:13.245058 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:13.341441 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:13.341364 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:11:13.342390 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:13.342369 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:11:17.491648 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:17.491550 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:21.895381 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:21.895192 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:26.393228 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:26.393175 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:30.787088 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:30.787031 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:31.850010 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:31.849977 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:32.705907 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:11:32.705867 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:11:35.286476 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:35.286446 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:39.717413 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:39.717378 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:44.213514 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:44.213478 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:48.682929 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:48.682895 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:53.177249 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:53.177217 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:11:57.620033 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:11:57.619987 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:02.706999 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:12:02.706956 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:12:03.582914 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:03.582874 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:06.500099 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:06.500040 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:08.907655 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:08.907624 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:11.221131 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:11.221094 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:13.927368 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:13.927328 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:15.791187 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:15.791150 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:20.277327 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:20.277279 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:21.153798 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:21.153725 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:21.339062 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:21.339032 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:24.685820 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:24.685762 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:28.601790 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:28.601739 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:30.013919 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:30.013881 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:30.277890 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:30.277804 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:31.073712 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:31.073667 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:32.170603 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:32.170551 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:32.707467 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:12:32.707430 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:12:36.841808 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:36.841751 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:37.089504 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:37.089467 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:37.115565 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:37.115486 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p"] Nov 02 07:12:37.115712 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:37.115682 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerName="istio-proxy" containerID="cri-o://5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838" gracePeriod=30 Nov 02 07:12:37.280500 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:37.280468 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:42.363799 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.363775 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:12:42.451895 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.451871 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-certs\") pod \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " Nov 02 07:12:42.452018 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.451910 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-kube-api-access-bm558\" (UniqueName: \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-kube-api-access-bm558\") pod \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " Nov 02 07:12:42.452018 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.451951 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-token\" (UniqueName: \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-token\") pod \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " Nov 02 07:12:42.452018 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.451972 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istiod-ca-cert\") pod \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " Nov 02 07:12:42.452018 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452005 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-podinfo\") pod \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " Nov 02 07:12:42.452226 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452067 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-credential-socket\") pod \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " Nov 02 07:12:42.452226 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452129 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-data\") pod \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " Nov 02 07:12:42.452226 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452132 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-certs" pod "eea2d1f7-93b6-4238-a3d2-8921d0d4784a" (UID: "eea2d1f7-93b6-4238-a3d2-8921d0d4784a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:12:42.452226 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452179 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-envoy\") pod \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " Nov 02 07:12:42.452226 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452216 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-socket\") pod \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\" (UID: \"eea2d1f7-93b6-4238-a3d2-8921d0d4784a\") " Nov 02 07:12:42.452494 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452373 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istiod-ca-cert" pod "eea2d1f7-93b6-4238-a3d2-8921d0d4784a" (UID: "eea2d1f7-93b6-4238-a3d2-8921d0d4784a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:12:42.452494 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452467 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-data" pod "eea2d1f7-93b6-4238-a3d2-8921d0d4784a" (UID: "eea2d1f7-93b6-4238-a3d2-8921d0d4784a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:12:42.452633 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452483 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-credential-socket" pod "eea2d1f7-93b6-4238-a3d2-8921d0d4784a" (UID: "eea2d1f7-93b6-4238-a3d2-8921d0d4784a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:12:42.452633 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452579 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-socket" pod "eea2d1f7-93b6-4238-a3d2-8921d0d4784a" (UID: "eea2d1f7-93b6-4238-a3d2-8921d0d4784a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:12:42.452749 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452714 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:42.452749 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452729 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:42.452859 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452749 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:42.452859 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452769 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:42.452859 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.452783 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:42.454198 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.454171 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-token" pod "eea2d1f7-93b6-4238-a3d2-8921d0d4784a" (UID: "eea2d1f7-93b6-4238-a3d2-8921d0d4784a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:12:42.454333 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.454292 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-envoy" pod "eea2d1f7-93b6-4238-a3d2-8921d0d4784a" (UID: "eea2d1f7-93b6-4238-a3d2-8921d0d4784a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:12:42.454570 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.454554 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-kube-api-access-bm558" pod "eea2d1f7-93b6-4238-a3d2-8921d0d4784a" (UID: "eea2d1f7-93b6-4238-a3d2-8921d0d4784a"). InnerVolumeSpecName "kube-api-access-bm558". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:12:42.454650 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.454630 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-podinfo" pod "eea2d1f7-93b6-4238-a3d2-8921d0d4784a" (UID: "eea2d1f7-93b6-4238-a3d2-8921d0d4784a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:12:42.554128 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.554099 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:42.554128 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.554129 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm558\" (UniqueName: \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-kube-api-access-bm558\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:42.554321 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.554140 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:42.554321 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.554148 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eea2d1f7-93b6-4238-a3d2-8921d0d4784a-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:12:42.923047 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.923015 5025 generic.go:356] "Generic (PLEG): container finished" podID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerID="5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838" exitCode=0 Nov 02 07:12:42.923261 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.923070 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" Nov 02 07:12:42.923261 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.923106 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" event={"ID":"eea2d1f7-93b6-4238-a3d2-8921d0d4784a","Type":"ContainerDied","Data":"5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838"} Nov 02 07:12:42.923261 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.923169 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p" event={"ID":"eea2d1f7-93b6-4238-a3d2-8921d0d4784a","Type":"ContainerDied","Data":"44342e069e27f79445f2ec22379eae86c5a43366ad891de9fa8ac0c35f53a595"} Nov 02 07:12:42.923261 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.923198 5025 scope.go:122] "RemoveContainer" containerID="5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838" Nov 02 07:12:42.932145 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.932120 5025 scope.go:122] "RemoveContainer" containerID="9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131" Nov 02 07:12:42.939860 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.939839 5025 scope.go:122] "RemoveContainer" containerID="5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838" Nov 02 07:12:42.940169 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:12:42.940152 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838\": container with ID starting with 5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838 not found: ID does not exist" containerID="5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838" Nov 02 07:12:42.940250 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.940178 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838"} err="failed to get container status \"5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838\": rpc error: code = NotFound desc = could not find container \"5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838\": container with ID starting with 5415f2ea4bb60e0f507a2ffd91668e1c739b2908728bad6052c8263a2b67c838 not found: ID does not exist" Nov 02 07:12:42.940250 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.940195 5025 scope.go:122] "RemoveContainer" containerID="9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131" Nov 02 07:12:42.940437 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:12:42.940420 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131\": container with ID starting with 9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131 not found: ID does not exist" containerID="9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131" Nov 02 07:12:42.940476 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:42.940441 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131"} err="failed to get container status \"9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131\": rpc error: code = NotFound desc = could not find container \"9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131\": container with ID starting with 9f0f00c41a150c3b59a81185db72c6493f6faa9674efda00c17c12b91a554131 not found: ID does not exist" Nov 02 07:12:43.081515 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:43.081486 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p"] Nov 02 07:12:43.129236 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:43.129205 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-614951-openshift-default-77cf6b545-lhp5p"] Nov 02 07:12:43.667910 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:43.667876 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" path="/var/lib/kubelet/pods/eea2d1f7-93b6-4238-a3d2-8921d0d4784a/volumes" Nov 02 07:12:51.255425 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:51.255393 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:51.545291 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:51.545202 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:12:57.176051 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:12:57.176014 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:13:02.706724 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:13:02.706685 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:13:32.522207 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:13:32.522173 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:13:32.706734 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:13:32.706698 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:14:02.707003 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:14:02.706966 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:14:32.706334 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:14:32.706255 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:14:32.784169 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:14:32.784137 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:02.787749 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:15:02.787700 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:15:18.866651 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:18.866605 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:23.596632 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:23.596565 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:24.023259 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.023224 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt"] Nov 02 07:15:24.023633 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.023621 5025 cpu_manager.go:414] "containerMap: removing container" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerName="istio-proxy" Nov 02 07:15:24.023680 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.023634 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerName="istio-proxy" Nov 02 07:15:24.023680 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.023646 5025 cpu_manager.go:414] "containerMap: removing container" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerName="istio-proxy" Nov 02 07:15:24.023680 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.023651 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerName="istio-proxy" Nov 02 07:15:24.023798 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.023701 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerName="istio-proxy" Nov 02 07:15:24.023798 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.023712 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="eea2d1f7-93b6-4238-a3d2-8921d0d4784a" containerName="istio-proxy" Nov 02 07:15:24.025526 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.025509 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.078405 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.078365 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-75b25b-openshift-default-dockercfg-b29hz\"" Nov 02 07:15:24.128927 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.128890 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt"] Nov 02 07:15:24.155364 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.155332 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00142986-2335-42cd-bea3-b8f803f17a25-istio-podinfo\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.155549 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.155390 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-istio-token\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.155549 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.155413 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-socket\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.155670 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.155556 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-certs\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.155670 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.155589 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-envoy\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.155747 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.155709 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z62jt\" (UniqueName: \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-kube-api-access-z62jt\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.155781 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.155757 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-credential-socket\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.155834 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.155805 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00142986-2335-42cd-bea3-b8f803f17a25-istiod-ca-cert\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.155910 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.155891 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-data\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.161867 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.161845 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:24.257335 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257292 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00142986-2335-42cd-bea3-b8f803f17a25-istio-podinfo\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.257544 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257363 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-istio-token\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.257544 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257385 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-socket\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.257544 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257403 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-certs\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.257544 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257419 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-envoy\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.257544 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257448 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z62jt\" (UniqueName: \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-kube-api-access-z62jt\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.257544 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257478 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-credential-socket\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.257544 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257521 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00142986-2335-42cd-bea3-b8f803f17a25-istiod-ca-cert\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.257940 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257563 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-data\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.257940 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257830 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-socket\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.257940 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257885 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-certs\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.258112 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.257986 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-credential-socket\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.258112 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.258006 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-data\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.258442 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.258421 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00142986-2335-42cd-bea3-b8f803f17a25-istiod-ca-cert\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.259741 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.259722 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-envoy\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.259833 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.259817 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00142986-2335-42cd-bea3-b8f803f17a25-istio-podinfo\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.323221 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.323106 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-istio-token\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.332123 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.332056 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z62jt\" (UniqueName: \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-kube-api-access-z62jt\") pod \"e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.336755 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.336716 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:24.541649 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.541596 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:24.764538 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.764500 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt"] Nov 02 07:15:24.800884 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:15:24.800855 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00142986_2335_42cd_bea3_b8f803f17a25.slice/crio-9fa74ebb8560845d11562cca6b9cc89be7666b5a184af024d6421e630bdaa0da WatchSource:0}: Error finding container 9fa74ebb8560845d11562cca6b9cc89be7666b5a184af024d6421e630bdaa0da: Status 404 returned error can't find the container with id 9fa74ebb8560845d11562cca6b9cc89be7666b5a184af024d6421e630bdaa0da Nov 02 07:15:24.802965 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.802933 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:15:24.803041 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.802998 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:15:24.803041 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.803028 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:15:24.813329 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:24.813298 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" event={"ID":"00142986-2335-42cd-bea3-b8f803f17a25","Type":"ContainerStarted","Data":"9fa74ebb8560845d11562cca6b9cc89be7666b5a184af024d6421e630bdaa0da"} Nov 02 07:15:25.819110 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:25.819059 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" event={"ID":"00142986-2335-42cd-bea3-b8f803f17a25","Type":"ContainerStarted","Data":"8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96"} Nov 02 07:15:25.980977 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:25.980921 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" podStartSLOduration=2.980906151 podStartE2EDuration="2.980906151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:15:25.979322138 +0000 UTC m=+3502.427698895" watchObservedRunningTime="2026-11-02 07:15:25.980906151 +0000 UTC m=+3502.429282987" Nov 02 07:15:26.338012 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:26.337973 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:26.343062 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:26.343036 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:26.823167 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:26.823134 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:26.824013 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:26.823992 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:28.891907 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:28.891877 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:29.072623 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:29.072588 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:32.042072 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:32.042034 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:32.225311 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:32.225278 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:32.707938 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:15:32.707900 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:15:33.150910 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:33.150834 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:34.724428 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:34.724394 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:34.908739 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:34.908699 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:35.100156 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:35.100034 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:39.369702 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:39.369654 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:43.973997 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:43.973903 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:48.587564 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:48.587527 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:51.952364 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:51.952277 5025 generic.go:356] "Generic (PLEG): container finished" podID="00142986-2335-42cd-bea3-b8f803f17a25" containerID="8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96" exitCode=0 Nov 02 07:15:51.952893 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:51.952356 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" event={"ID":"00142986-2335-42cd-bea3-b8f803f17a25","Type":"ContainerDied","Data":"8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96"} Nov 02 07:15:51.952893 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:51.952850 5025 scope.go:122] "RemoveContainer" containerID="8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96" Nov 02 07:15:51.954089 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:51.954043 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:15:51.954189 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:51.954127 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:15:51.954189 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:51.954161 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:15:52.958424 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:52.958389 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" event={"ID":"00142986-2335-42cd-bea3-b8f803f17a25","Type":"ContainerStarted","Data":"01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b"} Nov 02 07:15:53.337488 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:53.337404 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:53.338643 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:53.338619 5025 patch_prober.go:29] interesting pod/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 07:15:53.338754 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:53.338665 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 07:15:53.373850 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:53.372499 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:53.745975 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:53.745938 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:54.314962 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:54.314928 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:54.338290 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:54.338257 5025 patch_prober.go:29] interesting pod/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 07:15:54.338462 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:54.338312 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 07:15:54.508780 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:54.508728 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:55.337965 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:55.337931 5025 patch_prober.go:29] interesting pod/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 07:15:55.338423 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:55.337984 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 07:15:56.337830 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:56.337792 5025 patch_prober.go:29] interesting pod/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 07:15:56.338052 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:56.337856 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 07:15:56.823543 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:56.823506 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:57.337863 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:57.337826 5025 patch_prober.go:29] interesting pod/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Nov 02 07:15:57.338049 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:57.337889 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Nov 02 07:15:57.890894 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:57.890865 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:58.078502 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:58.078453 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:15:58.341743 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:58.341713 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:15:58.342533 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:15:58.342514 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:16:02.396930 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:02.396890 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:02.733475 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:16:02.733437 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:16:06.881744 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:06.881701 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:11.392024 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:11.387931 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:15.872609 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:15.872570 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:20.377968 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:20.377930 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:24.872695 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:24.872655 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:28.484880 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:28.484819 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:29.385201 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:29.384781 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:32.705453 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:16:32.705416 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:16:33.486417 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:33.486381 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:33.869768 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:33.869630 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:38.384716 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:38.384676 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:39.535730 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:39.535695 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:42.794348 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:42.794280 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:47.405270 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:47.405229 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:51.884195 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:51.884121 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:53.502154 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:53.502120 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:56.657901 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:56.657849 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:16:58.537695 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:16:58.537655 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:01.186422 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:01.186382 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:02.705817 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:17:02.705775 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:17:05.099533 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:05.099498 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:05.555667 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:05.555617 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:05.730663 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:05.730625 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:09.251053 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:09.251018 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:09.523797 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:09.523687 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:10.267311 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:10.267278 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:16.163241 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:16.163163 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:16.435732 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:16.435659 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:16.453579 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:16.453547 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt"] Nov 02 07:17:16.453803 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:16.453764 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" containerID="cri-o://01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b" gracePeriod=30 Nov 02 07:17:16.639397 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:16.639364 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:21.702569 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.702544 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:17:21.819386 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819318 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-kube-api-access-z62jt\" (UniqueName: \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-kube-api-access-z62jt\") pod \"00142986-2335-42cd-bea3-b8f803f17a25\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " Nov 02 07:17:21.819536 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819384 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-credential-socket\") pod \"00142986-2335-42cd-bea3-b8f803f17a25\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " Nov 02 07:17:21.819536 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819414 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-envoy\") pod \"00142986-2335-42cd-bea3-b8f803f17a25\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " Nov 02 07:17:21.819536 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819468 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-socket\") pod \"00142986-2335-42cd-bea3-b8f803f17a25\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " Nov 02 07:17:21.819536 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819519 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-istio-token\" (UniqueName: \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-istio-token\") pod \"00142986-2335-42cd-bea3-b8f803f17a25\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " Nov 02 07:17:21.819770 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819578 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00142986-2335-42cd-bea3-b8f803f17a25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00142986-2335-42cd-bea3-b8f803f17a25-istio-podinfo\") pod \"00142986-2335-42cd-bea3-b8f803f17a25\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " Nov 02 07:17:21.819770 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819608 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-data\") pod \"00142986-2335-42cd-bea3-b8f803f17a25\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " Nov 02 07:17:21.819770 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819641 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-certs\") pod \"00142986-2335-42cd-bea3-b8f803f17a25\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " Nov 02 07:17:21.819770 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819711 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00142986-2335-42cd-bea3-b8f803f17a25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00142986-2335-42cd-bea3-b8f803f17a25-istiod-ca-cert\") pod \"00142986-2335-42cd-bea3-b8f803f17a25\" (UID: \"00142986-2335-42cd-bea3-b8f803f17a25\") " Nov 02 07:17:21.819770 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819693 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-credential-socket" pod "00142986-2335-42cd-bea3-b8f803f17a25" (UID: "00142986-2335-42cd-bea3-b8f803f17a25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:17:21.819770 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819756 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-socket" pod "00142986-2335-42cd-bea3-b8f803f17a25" (UID: "00142986-2335-42cd-bea3-b8f803f17a25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:17:21.820127 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819917 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-data" pod "00142986-2335-42cd-bea3-b8f803f17a25" (UID: "00142986-2335-42cd-bea3-b8f803f17a25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:17:21.820127 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.819950 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-certs" pod "00142986-2335-42cd-bea3-b8f803f17a25" (UID: "00142986-2335-42cd-bea3-b8f803f17a25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:17:21.820127 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.820038 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:21.820127 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.820057 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:21.820349 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.820135 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:21.820349 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.820152 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:21.820349 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.820253 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00142986-2335-42cd-bea3-b8f803f17a25-istiod-ca-cert" pod "00142986-2335-42cd-bea3-b8f803f17a25" (UID: "00142986-2335-42cd-bea3-b8f803f17a25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:17:21.821674 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.821646 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-istio-token" pod "00142986-2335-42cd-bea3-b8f803f17a25" (UID: "00142986-2335-42cd-bea3-b8f803f17a25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:17:21.821674 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.821646 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-envoy" pod "00142986-2335-42cd-bea3-b8f803f17a25" (UID: "00142986-2335-42cd-bea3-b8f803f17a25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:17:21.821861 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.821700 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-kube-api-access-z62jt" pod "00142986-2335-42cd-bea3-b8f803f17a25" (UID: "00142986-2335-42cd-bea3-b8f803f17a25"). InnerVolumeSpecName "kube-api-access-z62jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:17:21.821861 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.821696 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00142986-2335-42cd-bea3-b8f803f17a25-istio-podinfo" pod "00142986-2335-42cd-bea3-b8f803f17a25" (UID: "00142986-2335-42cd-bea3-b8f803f17a25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:17:21.921262 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.921232 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:21.921262 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.921256 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00142986-2335-42cd-bea3-b8f803f17a25-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:21.921262 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.921266 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00142986-2335-42cd-bea3-b8f803f17a25-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:21.921491 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.921275 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z62jt\" (UniqueName: \"kubernetes.io/projected/00142986-2335-42cd-bea3-b8f803f17a25-kube-api-access-z62jt\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:21.921491 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:21.921284 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00142986-2335-42cd-bea3-b8f803f17a25-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:17:22.358026 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.357993 5025 generic.go:356] "Generic (PLEG): container finished" podID="00142986-2335-42cd-bea3-b8f803f17a25" containerID="01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b" exitCode=0 Nov 02 07:17:22.358207 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.358042 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" event={"ID":"00142986-2335-42cd-bea3-b8f803f17a25","Type":"ContainerDied","Data":"01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b"} Nov 02 07:17:22.358207 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.358106 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" event={"ID":"00142986-2335-42cd-bea3-b8f803f17a25","Type":"ContainerDied","Data":"9fa74ebb8560845d11562cca6b9cc89be7666b5a184af024d6421e630bdaa0da"} Nov 02 07:17:22.358207 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.358044 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt" Nov 02 07:17:22.358207 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.358132 5025 scope.go:122] "RemoveContainer" containerID="01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b" Nov 02 07:17:22.367552 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.367533 5025 scope.go:122] "RemoveContainer" containerID="8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96" Nov 02 07:17:22.375475 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.375457 5025 scope.go:122] "RemoveContainer" containerID="01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b" Nov 02 07:17:22.375721 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:17:22.375702 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b\": container with ID starting with 01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b not found: ID does not exist" containerID="01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b" Nov 02 07:17:22.375768 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.375731 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b"} err="failed to get container status \"01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b\": rpc error: code = NotFound desc = could not find container \"01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b\": container with ID starting with 01e268a34f8f4e8d2bbb4910dbab3e411a8b2c9497644f079dd9437517b9f24b not found: ID does not exist" Nov 02 07:17:22.375768 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.375749 5025 scope.go:122] "RemoveContainer" containerID="8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96" Nov 02 07:17:22.375956 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:17:22.375942 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96\": container with ID starting with 8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96 not found: ID does not exist" containerID="8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96" Nov 02 07:17:22.375996 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.375962 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96"} err="failed to get container status \"8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96\": rpc error: code = NotFound desc = could not find container \"8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96\": container with ID starting with 8e6e6fa2b8e9f6d00554a09f2fe4f001e5ac44c14fa00908b29b92649204bc96 not found: ID does not exist" Nov 02 07:17:22.562711 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.562678 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt"] Nov 02 07:17:22.610305 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:22.610239 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-75b25b-openshift-default-66856fd4454tczt"] Nov 02 07:17:23.565014 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:23.564977 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:23.667542 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:23.667503 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00142986-2335-42cd-bea3-b8f803f17a25" path="/var/lib/kubelet/pods/00142986-2335-42cd-bea3-b8f803f17a25/volumes" Nov 02 07:17:32.708927 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:17:32.708887 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:17:33.875318 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:33.875279 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:35.144609 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:35.144579 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:36.039567 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:36.039512 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:17:50.342923 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:17:50.342889 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:18:02.706814 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:18:02.706778 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:18:03.252057 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:03.252020 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:18:11.073249 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:11.073197 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-fmss9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:18:11.074013 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:11.073962 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-fmss9" podUID="24a7b3c3-fb09-4b70-aa73-858f52476799" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:18:11.249318 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:11.249272 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:18:11.249612 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:11.249580 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-nsgcs" podUID="9b4f70ef-1722-4b66-9a44-5c771b7a24b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:18:11.360642 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:11.360497 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:18:11.360951 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:11.360898 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-8xsqm" podUID="2c57c5a3-8442-4aa2-967c-a9a80aa58d7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:18:32.009116 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:32.009086 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:18:32.182865 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:32.182827 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:18:32.706305 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:18:32.706264 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:18:34.173222 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:34.173189 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:18:47.220134 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:47.220022 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:18:58.826060 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:18:58.826030 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:19:02.708174 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:19:02.708129 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:19:32.707243 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:19:32.707204 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:19:34.516320 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:34.516287 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:19:55.469853 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:55.469818 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:19:56.016307 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.016268 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7"] Nov 02 07:19:56.016731 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.016715 5025 cpu_manager.go:414] "containerMap: removing container" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" Nov 02 07:19:56.016779 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.016732 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" Nov 02 07:19:56.016779 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.016741 5025 cpu_manager.go:414] "containerMap: removing container" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" Nov 02 07:19:56.016779 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.016747 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" Nov 02 07:19:56.016879 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.016805 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" Nov 02 07:19:56.019857 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.019839 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.053540 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.053504 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tmpg\" (UniqueName: \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-kube-api-access-6tmpg\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.053721 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.053547 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d7149a-760d-4214-9467-0e19bb2e98fe-istiod-ca-cert\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.053721 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.053566 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-token\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.053721 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.053695 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-data\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.053913 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.053835 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-envoy\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.053913 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.053865 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-credential-socket\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.053981 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.053939 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-podinfo\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.053981 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.053967 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-socket\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.054053 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.053985 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-certs\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.089701 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.089677 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0e3c0bf7-openshift-default-dockercfg-qkmmm\"" Nov 02 07:19:56.094793 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.094765 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7"] Nov 02 07:19:56.111333 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.111286 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:19:56.154839 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.154800 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-podinfo\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.154839 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.154840 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-socket\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.154856 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-certs\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.154909 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tmpg\" (UniqueName: \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-kube-api-access-6tmpg\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.154933 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d7149a-760d-4214-9467-0e19bb2e98fe-istiod-ca-cert\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.154952 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-token\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.154985 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-data\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.155089 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-envoy\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155138 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.155114 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-credential-socket\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155499 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.155347 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-socket\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155499 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.155367 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-data\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155499 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.155449 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-credential-socket\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155704 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.155684 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d7149a-760d-4214-9467-0e19bb2e98fe-istiod-ca-cert\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.155773 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.155715 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-certs\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.157367 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.157342 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-envoy\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.157472 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.157387 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-podinfo\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.216923 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.216882 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-token\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.220770 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.220730 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tmpg\" (UniqueName: \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-kube-api-access-6tmpg\") pod \"e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.317357 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.317269 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:19:56.333251 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.333220 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:56.746016 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.745988 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7"] Nov 02 07:19:56.747153 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:19:56.747128 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d7149a_760d_4214_9467_0e19bb2e98fe.slice/crio-7f5579309aeb7acfdf9ef649dad730d62c0aa305d3df9b7826a393758532ee43 WatchSource:0}: Error finding container 7f5579309aeb7acfdf9ef649dad730d62c0aa305d3df9b7826a393758532ee43: Status 404 returned error can't find the container with id 7f5579309aeb7acfdf9ef649dad730d62c0aa305d3df9b7826a393758532ee43 Nov 02 07:19:56.749188 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.749150 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:19:56.749288 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.749218 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:19:56.749288 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:56.749250 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:19:57.177104 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.177043 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" event={"ID":"18d7149a-760d-4214-9467-0e19bb2e98fe","Type":"ContainerStarted","Data":"53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709"} Nov 02 07:19:57.177271 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.177125 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" event={"ID":"18d7149a-760d-4214-9467-0e19bb2e98fe","Type":"ContainerStarted","Data":"7f5579309aeb7acfdf9ef649dad730d62c0aa305d3df9b7826a393758532ee43"} Nov 02 07:19:57.223735 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.223708 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2"] Nov 02 07:19:57.224187 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.224172 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="00142986-2335-42cd-bea3-b8f803f17a25" containerName="istio-proxy" Nov 02 07:19:57.227039 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.227025 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.267193 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.267159 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-maas-api-tls\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.267353 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.267237 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-metrics-certs\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.267353 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.267261 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9h4d\" (UniqueName: \"kubernetes.io/projected/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-kube-api-access-l9h4d\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.267353 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.267301 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-openshift-service-ca\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.280720 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.280675 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2"] Nov 02 07:19:57.333107 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.333039 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" podStartSLOduration=2.333025458 podStartE2EDuration="2.333025458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:19:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:19:57.331507578 +0000 UTC m=+3773.779884352" watchObservedRunningTime="2026-11-02 07:19:57.333025458 +0000 UTC m=+3773.781402224" Nov 02 07:19:57.333937 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.333910 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:57.338682 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.338662 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:57.368369 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.368343 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-maas-api-tls\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.368459 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.368414 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-metrics-certs\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.368459 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.368437 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9h4d\" (UniqueName: \"kubernetes.io/projected/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-kube-api-access-l9h4d\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.368594 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.368463 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-openshift-service-ca\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.369164 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.369139 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-openshift-service-ca\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.370894 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.370875 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-metrics-certs\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.370954 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.370934 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-maas-api-tls\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.422494 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.422463 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9h4d\" (UniqueName: \"kubernetes.io/projected/d5028bed-1e85-4b46-80c1-ca4d9b92d1c2-kube-api-access-l9h4d\") pod \"maas-api-5fbfc94f97-2dwj2\" (UID: \"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2\") " pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.540098 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.539984 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:19:57.907130 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.907100 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2"] Nov 02 07:19:57.908363 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:19:57.908333 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5028bed_1e85_4b46_80c1_ca4d9b92d1c2.slice/crio-5e717c591181e4f252dd373e0a10ba091fb0ed565091db28eeed96e216827dbb WatchSource:0}: Error finding container 5e717c591181e4f252dd373e0a10ba091fb0ed565091db28eeed96e216827dbb: Status 404 returned error can't find the container with id 5e717c591181e4f252dd373e0a10ba091fb0ed565091db28eeed96e216827dbb Nov 02 07:19:57.909734 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:57.909711 5025 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 02 07:19:58.182415 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:58.182374 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" event={"ID":"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2","Type":"ContainerStarted","Data":"5e717c591181e4f252dd373e0a10ba091fb0ed565091db28eeed96e216827dbb"} Nov 02 07:19:58.182624 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:58.182604 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:58.183996 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:58.183974 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:19:59.188280 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:59.188245 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" event={"ID":"d5028bed-1e85-4b46-80c1-ca4d9b92d1c2","Type":"ContainerStarted","Data":"a3dc3ffecccf3f3f2362a2b220e6236202fe9ab0ac2b277bfb1633554a072574"} Nov 02 07:19:59.298862 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:59.298817 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" podStartSLOduration=1.443509471 podStartE2EDuration="2.298803913s" totalImagesPullingTime="855.294442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:19:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:19:59.298292336 +0000 UTC m=+3775.746669106" watchObservedRunningTime="2026-11-02 07:19:59.298803913 +0000 UTC m=+3775.747180666" Nov 02 07:19:59.510971 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:59.510883 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:19:59.704618 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:19:59.704564 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:00.192656 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:00.192620 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:20:02.615102 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:02.605995 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:02.706499 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:20:02.706461 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:20:02.796560 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:02.796521 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:03.144933 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:03.144812 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:03.346031 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:03.346000 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:03.574603 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:03.574575 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:06.202768 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:06.202739 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fbfc94f97-2dwj2" Nov 02 07:20:06.361733 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:06.361700 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj"] Nov 02 07:20:06.361975 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:06.361952 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" podUID="ed97b347-05b9-4efe-b1d4-2427cf047d03" containerName="maas-api" containerID="cri-o://5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9" gracePeriod=30 Nov 02 07:20:07.619108 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.619067 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 07:20:07.687301 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.687229 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed97b347-05b9-4efe-b1d4-2427cf047d03-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed97b347-05b9-4efe-b1d4-2427cf047d03-openshift-service-ca\") pod \"ed97b347-05b9-4efe-b1d4-2427cf047d03\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " Nov 02 07:20:07.687466 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.687372 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-maas-api-tls\") pod \"ed97b347-05b9-4efe-b1d4-2427cf047d03\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " Nov 02 07:20:07.687466 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.687416 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-metrics-certs\") pod \"ed97b347-05b9-4efe-b1d4-2427cf047d03\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " Nov 02 07:20:07.687466 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.687439 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed97b347-05b9-4efe-b1d4-2427cf047d03-kube-api-access-h29j4\" (UniqueName: \"kubernetes.io/projected/ed97b347-05b9-4efe-b1d4-2427cf047d03-kube-api-access-h29j4\") pod \"ed97b347-05b9-4efe-b1d4-2427cf047d03\" (UID: \"ed97b347-05b9-4efe-b1d4-2427cf047d03\") " Nov 02 07:20:07.687645 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.687618 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed97b347-05b9-4efe-b1d4-2427cf047d03-openshift-service-ca" pod "ed97b347-05b9-4efe-b1d4-2427cf047d03" (UID: "ed97b347-05b9-4efe-b1d4-2427cf047d03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:20:07.687748 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.687717 5025 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed97b347-05b9-4efe-b1d4-2427cf047d03-openshift-service-ca\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:07.689533 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.689507 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-metrics-certs" pod "ed97b347-05b9-4efe-b1d4-2427cf047d03" (UID: "ed97b347-05b9-4efe-b1d4-2427cf047d03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:20:07.689675 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.689603 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-maas-api-tls" pod "ed97b347-05b9-4efe-b1d4-2427cf047d03" (UID: "ed97b347-05b9-4efe-b1d4-2427cf047d03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 02 07:20:07.689675 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.689607 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed97b347-05b9-4efe-b1d4-2427cf047d03-kube-api-access-h29j4" pod "ed97b347-05b9-4efe-b1d4-2427cf047d03" (UID: "ed97b347-05b9-4efe-b1d4-2427cf047d03"). InnerVolumeSpecName "kube-api-access-h29j4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:20:07.788412 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.788377 5025 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-metrics-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:07.788412 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.788408 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h29j4\" (UniqueName: \"kubernetes.io/projected/ed97b347-05b9-4efe-b1d4-2427cf047d03-kube-api-access-h29j4\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:07.788412 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:07.788418 5025 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ed97b347-05b9-4efe-b1d4-2427cf047d03-maas-api-tls\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:08.165792 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:08.165679 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:08.228156 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:08.228122 5025 generic.go:356] "Generic (PLEG): container finished" podID="ed97b347-05b9-4efe-b1d4-2427cf047d03" containerID="5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9" exitCode=0 Nov 02 07:20:08.228332 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:08.228176 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" Nov 02 07:20:08.228332 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:08.228203 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" event={"ID":"ed97b347-05b9-4efe-b1d4-2427cf047d03","Type":"ContainerDied","Data":"5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9"} Nov 02 07:20:08.228332 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:08.228259 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj" event={"ID":"ed97b347-05b9-4efe-b1d4-2427cf047d03","Type":"ContainerDied","Data":"4a5ef395d9f63b5743cd0948c9137f3d20a645cc3ec38c9889451d44e674a493"} Nov 02 07:20:08.228332 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:08.228282 5025 scope.go:122] "RemoveContainer" containerID="5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9" Nov 02 07:20:08.238172 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:08.238144 5025 scope.go:122] "RemoveContainer" containerID="5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9" Nov 02 07:20:08.238416 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:20:08.238396 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9\": container with ID starting with 5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9 not found: ID does not exist" containerID="5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9" Nov 02 07:20:08.238466 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:08.238424 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9"} err="failed to get container status \"5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9\": rpc error: code = NotFound desc = could not find container \"5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9\": container with ID starting with 5f7e89b46c0cb076f3d11f1ecc3a8c9d6fc9c38662caa7dcdf2e7c799fcdeef9 not found: ID does not exist" Nov 02 07:20:08.384195 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:08.384161 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj"] Nov 02 07:20:08.432429 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:08.432353 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7dc858b877-qxvmj"] Nov 02 07:20:09.668128 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:09.668097 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed97b347-05b9-4efe-b1d4-2427cf047d03" path="/var/lib/kubelet/pods/ed97b347-05b9-4efe-b1d4-2427cf047d03/volumes" Nov 02 07:20:12.690098 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:12.687623 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:17.093162 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:17.092677 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:21.292754 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:21.292722 5025 generic.go:356] "Generic (PLEG): container finished" podID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerID="53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709" exitCode=0 Nov 02 07:20:21.293213 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:21.292777 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" event={"ID":"18d7149a-760d-4214-9467-0e19bb2e98fe","Type":"ContainerDied","Data":"53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709"} Nov 02 07:20:21.293213 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:21.293117 5025 scope.go:122] "RemoveContainer" containerID="53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709" Nov 02 07:20:21.294261 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:21.294225 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:20:21.294358 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:21.294315 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:20:21.294358 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:21.294347 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:20:21.918097 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:21.916619 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:22.311867 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:22.311782 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" event={"ID":"18d7149a-760d-4214-9467-0e19bb2e98fe","Type":"ContainerStarted","Data":"64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf"} Nov 02 07:20:22.334257 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:22.334229 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:20:22.334565 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:22.334543 5025 patch_prober.go:29] interesting pod/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 02 07:20:22.334631 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:22.334614 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" podUID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 02 07:20:23.333759 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:23.333731 5025 patch_prober.go:29] interesting pod/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 02 07:20:23.334201 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:23.333786 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" podUID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 02 07:20:24.333735 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:24.333702 5025 patch_prober.go:29] interesting pod/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 02 07:20:24.333925 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:24.333766 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" podUID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 02 07:20:25.333994 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:25.333962 5025 patch_prober.go:29] interesting pod/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 02 07:20:25.334427 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:25.334008 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" podUID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 02 07:20:26.334013 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:26.333983 5025 patch_prober.go:29] interesting pod/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" start-of-body= Nov 02 07:20:26.334422 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:26.334032 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" podUID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.150:15021/healthz/ready\": dial tcp 10.133.0.150:15021: connect: connection refused" Nov 02 07:20:26.592180 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:26.583335 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:27.338245 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:27.338218 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:20:27.338635 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:27.338488 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:20:27.339037 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:27.339021 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:20:28.434792 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:28.434761 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:31.076175 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:31.076100 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:32.706727 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:20:32.706688 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:20:34.812989 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:34.812952 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:35.490848 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:35.490813 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:39.977675 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:39.977625 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:42.543448 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:42.542663 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:42.723678 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:42.723644 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:44.360445 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:44.360400 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:48.274145 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:48.274107 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:49.666794 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:49.666722 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:49.920559 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:49.920468 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:51.928301 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:51.928268 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7"] Nov 02 07:20:51.928748 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:51.928503 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" podUID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerName="istio-proxy" containerID="cri-o://64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf" gracePeriod=30 Nov 02 07:20:51.967271 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:51.967246 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:53.178022 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.177990 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:53.657025 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.656920 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp"] Nov 02 07:20:53.657391 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.657377 5025 cpu_manager.go:414] "containerMap: removing container" podUID="ed97b347-05b9-4efe-b1d4-2427cf047d03" containerName="maas-api" Nov 02 07:20:53.657449 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.657392 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed97b347-05b9-4efe-b1d4-2427cf047d03" containerName="maas-api" Nov 02 07:20:53.657485 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.657448 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed97b347-05b9-4efe-b1d4-2427cf047d03" containerName="maas-api" Nov 02 07:20:53.660428 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.660411 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.710495 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.710456 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-c0d3ab-openshift-default-dockercfg-9r47t\"" Nov 02 07:20:53.760171 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.760136 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp"] Nov 02 07:20:53.779576 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.779547 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-socket\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.779759 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.779591 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-credential-socket\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.779759 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.779614 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-envoy\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.779759 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.779723 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-certs\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.779759 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.779747 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a312903-03e0-43d4-8240-a8e60cdaedca-istiod-ca-cert\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.779986 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.779857 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-token\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.779986 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.779916 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-podinfo\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.779986 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.779956 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-data\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.780164 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.780102 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4869r\" (UniqueName: \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-kube-api-access-4869r\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.808150 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.808102 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:53.880663 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.880629 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4869r\" (UniqueName: \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-kube-api-access-4869r\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.880875 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.880676 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-socket\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.880875 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.880700 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-credential-socket\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.880875 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.880729 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-envoy\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.880875 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.880753 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-certs\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.880875 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.880776 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a312903-03e0-43d4-8240-a8e60cdaedca-istiod-ca-cert\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.880875 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.880812 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-token\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.880875 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.880839 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-podinfo\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.880875 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.880860 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-data\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.881366 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.881148 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-credential-socket\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.881366 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.881217 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-socket\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.881366 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.881252 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-certs\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.881366 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.881314 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-data\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.881600 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.881582 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a312903-03e0-43d4-8240-a8e60cdaedca-istiod-ca-cert\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.883302 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.883279 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-envoy\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.883507 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.883491 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-podinfo\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.935265 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.935227 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-token\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.935415 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.935291 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4869r\" (UniqueName: \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-kube-api-access-4869r\") pod \"e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:53.973532 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:53.973496 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:54.105268 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:54.105223 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:54.386850 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:54.386755 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp"] Nov 02 07:20:54.421718 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:20:54.421681 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a312903_03e0_43d4_8240_a8e60cdaedca.slice/crio-996e7168ff5bb02c493b5866922f29021db09177d3c876da8fe5f51e56c81a7f WatchSource:0}: Error finding container 996e7168ff5bb02c493b5866922f29021db09177d3c876da8fe5f51e56c81a7f: Status 404 returned error can't find the container with id 996e7168ff5bb02c493b5866922f29021db09177d3c876da8fe5f51e56c81a7f Nov 02 07:20:54.423852 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:54.423818 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:20:54.423916 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:54.423883 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:20:54.423961 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:54.423913 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:20:54.450665 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:54.450635 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" event={"ID":"3a312903-03e0-43d4-8240-a8e60cdaedca","Type":"ContainerStarted","Data":"996e7168ff5bb02c493b5866922f29021db09177d3c876da8fe5f51e56c81a7f"} Nov 02 07:20:55.456316 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:55.456273 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" event={"ID":"3a312903-03e0-43d4-8240-a8e60cdaedca","Type":"ContainerStarted","Data":"2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da"} Nov 02 07:20:55.567717 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:55.567668 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" podStartSLOduration=2.5676557190000002 podStartE2EDuration="2.567655719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 07:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:20:55.56660954 +0000 UTC m=+3832.014986295" watchObservedRunningTime="2026-11-02 07:20:55.567655719 +0000 UTC m=+3832.016032474" Nov 02 07:20:55.974466 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:55.974431 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:55.979638 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:55.979609 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:56.461102 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:56.461063 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:56.462099 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:56.462063 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:20:57.177625 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.177600 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:20:57.316848 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.316771 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d7149a-760d-4214-9467-0e19bb2e98fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d7149a-760d-4214-9467-0e19bb2e98fe-istiod-ca-cert\") pod \"18d7149a-760d-4214-9467-0e19bb2e98fe\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " Nov 02 07:20:57.316848 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.316813 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-token\" (UniqueName: \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-token\") pod \"18d7149a-760d-4214-9467-0e19bb2e98fe\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " Nov 02 07:20:57.317036 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.316850 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-data\") pod \"18d7149a-760d-4214-9467-0e19bb2e98fe\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " Nov 02 07:20:57.317036 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.316916 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-socket\") pod \"18d7149a-760d-4214-9467-0e19bb2e98fe\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " Nov 02 07:20:57.317036 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.316969 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-envoy\") pod \"18d7149a-760d-4214-9467-0e19bb2e98fe\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " Nov 02 07:20:57.317036 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317008 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-credential-socket\") pod \"18d7149a-760d-4214-9467-0e19bb2e98fe\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " Nov 02 07:20:57.317036 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317037 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-podinfo\") pod \"18d7149a-760d-4214-9467-0e19bb2e98fe\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " Nov 02 07:20:57.317332 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317110 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-kube-api-access-6tmpg\" (UniqueName: \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-kube-api-access-6tmpg\") pod \"18d7149a-760d-4214-9467-0e19bb2e98fe\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " Nov 02 07:20:57.317332 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317142 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-certs\") pod \"18d7149a-760d-4214-9467-0e19bb2e98fe\" (UID: \"18d7149a-760d-4214-9467-0e19bb2e98fe\") " Nov 02 07:20:57.317332 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317175 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d7149a-760d-4214-9467-0e19bb2e98fe-istiod-ca-cert" pod "18d7149a-760d-4214-9467-0e19bb2e98fe" (UID: "18d7149a-760d-4214-9467-0e19bb2e98fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:20:57.317511 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317318 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-socket" pod "18d7149a-760d-4214-9467-0e19bb2e98fe" (UID: "18d7149a-760d-4214-9467-0e19bb2e98fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:20:57.317511 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317400 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-credential-socket" pod "18d7149a-760d-4214-9467-0e19bb2e98fe" (UID: "18d7149a-760d-4214-9467-0e19bb2e98fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:20:57.317629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317571 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-certs" pod "18d7149a-760d-4214-9467-0e19bb2e98fe" (UID: "18d7149a-760d-4214-9467-0e19bb2e98fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:20:57.317629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317581 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-data" pod "18d7149a-760d-4214-9467-0e19bb2e98fe" (UID: "18d7149a-760d-4214-9467-0e19bb2e98fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:20:57.317629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317576 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18d7149a-760d-4214-9467-0e19bb2e98fe-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:57.317629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317613 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:57.317629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.317626 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:57.319279 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.319253 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-podinfo" pod "18d7149a-760d-4214-9467-0e19bb2e98fe" (UID: "18d7149a-760d-4214-9467-0e19bb2e98fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:20:57.319369 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.319296 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-kube-api-access-6tmpg" pod "18d7149a-760d-4214-9467-0e19bb2e98fe" (UID: "18d7149a-760d-4214-9467-0e19bb2e98fe"). InnerVolumeSpecName "kube-api-access-6tmpg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:20:57.319550 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.319522 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-envoy" pod "18d7149a-760d-4214-9467-0e19bb2e98fe" (UID: "18d7149a-760d-4214-9467-0e19bb2e98fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:20:57.319550 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.319536 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-token" pod "18d7149a-760d-4214-9467-0e19bb2e98fe" (UID: "18d7149a-760d-4214-9467-0e19bb2e98fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:20:57.419135 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.419109 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:57.419135 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.419134 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tmpg\" (UniqueName: \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-kube-api-access-6tmpg\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:57.419305 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.419145 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:57.419305 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.419154 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:57.419305 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.419163 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:57.419305 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.419170 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18d7149a-760d-4214-9467-0e19bb2e98fe-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:20:57.466865 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.466830 5025 generic.go:356] "Generic (PLEG): container finished" podID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerID="64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf" exitCode=0 Nov 02 07:20:57.467334 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.466881 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" Nov 02 07:20:57.467334 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.466914 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" event={"ID":"18d7149a-760d-4214-9467-0e19bb2e98fe","Type":"ContainerDied","Data":"64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf"} Nov 02 07:20:57.467334 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.466980 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7" event={"ID":"18d7149a-760d-4214-9467-0e19bb2e98fe","Type":"ContainerDied","Data":"7f5579309aeb7acfdf9ef649dad730d62c0aa305d3df9b7826a393758532ee43"} Nov 02 07:20:57.467334 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.467027 5025 scope.go:122] "RemoveContainer" containerID="64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf" Nov 02 07:20:57.476136 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.476120 5025 scope.go:122] "RemoveContainer" containerID="53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709" Nov 02 07:20:57.484109 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.484091 5025 scope.go:122] "RemoveContainer" containerID="64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf" Nov 02 07:20:57.484363 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:20:57.484345 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf\": container with ID starting with 64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf not found: ID does not exist" containerID="64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf" Nov 02 07:20:57.484442 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.484375 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf"} err="failed to get container status \"64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf\": rpc error: code = NotFound desc = could not find container \"64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf\": container with ID starting with 64076b3f8e2528ffd730f1730eba16b9f285be272deb37cbf6b1f7ea08407dbf not found: ID does not exist" Nov 02 07:20:57.484442 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.484399 5025 scope.go:122] "RemoveContainer" containerID="53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709" Nov 02 07:20:57.484668 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:20:57.484645 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709\": container with ID starting with 53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709 not found: ID does not exist" containerID="53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709" Nov 02 07:20:57.484712 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.484676 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709"} err="failed to get container status \"53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709\": rpc error: code = NotFound desc = could not find container \"53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709\": container with ID starting with 53c9e383bec5d63bf50fdb17bcf7cc768381d7653f5b565026e41f4f4fa42709 not found: ID does not exist" Nov 02 07:20:57.623398 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.623322 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7"] Nov 02 07:20:57.626214 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.626188 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0e3c0bf7-openshift-default-768df6d985-7wmb7"] Nov 02 07:20:57.668207 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:57.668181 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d7149a-760d-4214-9467-0e19bb2e98fe" path="/var/lib/kubelet/pods/18d7149a-760d-4214-9467-0e19bb2e98fe/volumes" Nov 02 07:20:58.347352 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:58.347317 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:20:58.533734 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:20:58.533698 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:01.448428 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:01.448388 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:01.630582 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:01.630538 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:02.706991 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:21:02.706955 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:21:04.227989 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:04.227947 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:04.402160 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:04.402120 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:08.872941 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:08.872889 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:13.388104 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:13.383912 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:17.775316 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:17.775283 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:21.578444 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:21.578407 5025 generic.go:356] "Generic (PLEG): container finished" podID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerID="2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da" exitCode=0 Nov 02 07:21:21.578910 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:21.578467 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" event={"ID":"3a312903-03e0-43d4-8240-a8e60cdaedca","Type":"ContainerDied","Data":"2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da"} Nov 02 07:21:21.578981 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:21.578943 5025 scope.go:122] "RemoveContainer" containerID="2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da" Nov 02 07:21:21.580185 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:21.580147 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:21:21.580254 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:21.580229 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:21:21.580291 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:21.580261 5025 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 02 07:21:22.495097 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:22.483847 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:22.598103 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:22.597730 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" event={"ID":"3a312903-03e0-43d4-8240-a8e60cdaedca","Type":"ContainerStarted","Data":"4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813"} Nov 02 07:21:22.974549 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:22.974511 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:21:22.975901 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:22.975872 5025 patch_prober.go:29] interesting pod/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 07:21:22.976030 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:22.975927 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 07:21:23.231147 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:23.228717 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:23.974855 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:23.974818 5025 patch_prober.go:29] interesting pod/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 07:21:23.975284 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:23.974872 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 07:21:24.974490 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:24.974450 5025 patch_prober.go:29] interesting pod/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 07:21:24.974677 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:24.974503 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 07:21:25.974415 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:25.974373 5025 patch_prober.go:29] interesting pod/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 07:21:25.974879 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:25.974433 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 07:21:26.461282 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:26.461235 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:21:26.871767 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:26.871673 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:26.974845 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:26.974806 5025 patch_prober.go:29] interesting pod/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Nov 02 07:21:26.975306 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:26.974875 5025 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Nov 02 07:21:27.978658 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:27.978628 5025 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:21:27.979567 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:27.979544 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:21:30.536301 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:30.536270 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:31.290918 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:31.290868 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:32.707191 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:21:32.707147 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:21:35.143402 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:35.143371 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:35.684479 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:35.684400 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:40.118160 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:40.118123 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:43.784219 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:43.784188 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:43.974180 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:43.974105 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:44.471624 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:44.471585 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:48.875966 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:48.875929 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:53.292066 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:53.292031 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:21:57.671148 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:21:57.671111 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:02.062110 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:02.061755 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:02.706405 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:22:02.706364 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:22:06.482273 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:06.482233 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:10.881160 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:10.881119 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:15.287408 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:15.287354 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:19.682686 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:19.682650 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:24.173259 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:24.173212 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:28.573095 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:28.572852 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:32.705664 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:22:32.705617 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:22:32.983380 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:32.983209 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:35.533057 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:35.533001 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:37.370814 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:37.370760 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:41.765986 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:41.765950 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:44.154320 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:44.154290 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:44.338977 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:44.338944 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:46.482503 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:46.482468 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:51.123204 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:51.123168 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:55.249323 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:55.249283 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:55.448542 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:55.448510 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:22:55.624191 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:22:55.624097 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:02.706467 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:23:02.706429 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:23:04.390437 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:04.390394 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:05.775387 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:05.775357 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:06.042151 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:06.042042 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:06.772261 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:06.772212 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:12.509969 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:12.509512 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:12.695280 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:12.695247 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:12.788783 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:12.788706 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp"] Nov 02 07:23:12.788936 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:12.788913 5025 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerName="istio-proxy" containerID="cri-o://4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813" gracePeriod=30 Nov 02 07:23:12.873106 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:12.870425 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:12.981349 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:12.981305 5025 patch_prober.go:29] interesting pod/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 02 07:23:12.981522 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:12.981384 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 02 07:23:18.102681 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.102650 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:23:18.132147 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.132115 5025 generic.go:356] "Generic (PLEG): container finished" podID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerID="4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813" exitCode=0 Nov 02 07:23:18.132322 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.132180 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" event={"ID":"3a312903-03e0-43d4-8240-a8e60cdaedca","Type":"ContainerDied","Data":"4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813"} Nov 02 07:23:18.132322 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.132232 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" event={"ID":"3a312903-03e0-43d4-8240-a8e60cdaedca","Type":"ContainerDied","Data":"996e7168ff5bb02c493b5866922f29021db09177d3c876da8fe5f51e56c81a7f"} Nov 02 07:23:18.132322 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.132260 5025 scope.go:122] "RemoveContainer" containerID="4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813" Nov 02 07:23:18.132505 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.132414 5025 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp" Nov 02 07:23:18.142928 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.142905 5025 scope.go:122] "RemoveContainer" containerID="2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da" Nov 02 07:23:18.152433 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.152398 5025 scope.go:122] "RemoveContainer" containerID="4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813" Nov 02 07:23:18.152757 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:23:18.152735 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813\": container with ID starting with 4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813 not found: ID does not exist" containerID="4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813" Nov 02 07:23:18.152860 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.152775 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813"} err="failed to get container status \"4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813\": rpc error: code = NotFound desc = could not find container \"4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813\": container with ID starting with 4c22f740a5fb36d42f4cc9c78e5a98966282186d303a586dac421ae1593f4813 not found: ID does not exist" Nov 02 07:23:18.152860 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.152795 5025 scope.go:122] "RemoveContainer" containerID="2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da" Nov 02 07:23:18.153106 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:23:18.153067 5025 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da\": container with ID starting with 2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da not found: ID does not exist" containerID="2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da" Nov 02 07:23:18.153209 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.153106 5025 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da"} err="failed to get container status \"2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da\": rpc error: code = NotFound desc = could not find container \"2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da\": container with ID starting with 2b5b693b94d75e7b778ebc1a65c0f6724141fe0cfa909d3085021911063544da not found: ID does not exist" Nov 02 07:23:18.274363 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.274322 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-socket\") pod \"3a312903-03e0-43d4-8240-a8e60cdaedca\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " Nov 02 07:23:18.274555 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.274393 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-credential-socket\") pod \"3a312903-03e0-43d4-8240-a8e60cdaedca\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " Nov 02 07:23:18.274555 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.274458 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-kube-api-access-4869r\" (UniqueName: \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-kube-api-access-4869r\") pod \"3a312903-03e0-43d4-8240-a8e60cdaedca\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " Nov 02 07:23:18.274555 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.274519 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-certs\") pod \"3a312903-03e0-43d4-8240-a8e60cdaedca\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " Nov 02 07:23:18.274764 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.274583 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-podinfo\") pod \"3a312903-03e0-43d4-8240-a8e60cdaedca\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " Nov 02 07:23:18.274764 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.274619 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a312903-03e0-43d4-8240-a8e60cdaedca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a312903-03e0-43d4-8240-a8e60cdaedca-istiod-ca-cert\") pod \"3a312903-03e0-43d4-8240-a8e60cdaedca\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " Nov 02 07:23:18.274764 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.274659 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-data\") pod \"3a312903-03e0-43d4-8240-a8e60cdaedca\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " Nov 02 07:23:18.274764 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.274699 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-envoy\") pod \"3a312903-03e0-43d4-8240-a8e60cdaedca\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " Nov 02 07:23:18.274764 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.274750 5025 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-token\" (UniqueName: \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-token\") pod \"3a312903-03e0-43d4-8240-a8e60cdaedca\" (UID: \"3a312903-03e0-43d4-8240-a8e60cdaedca\") " Nov 02 07:23:18.276239 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.275935 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-data" pod "3a312903-03e0-43d4-8240-a8e60cdaedca" (UID: "3a312903-03e0-43d4-8240-a8e60cdaedca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:23:18.276239 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.275964 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a312903-03e0-43d4-8240-a8e60cdaedca-istiod-ca-cert" pod "3a312903-03e0-43d4-8240-a8e60cdaedca" (UID: "3a312903-03e0-43d4-8240-a8e60cdaedca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 02 07:23:18.276239 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.276120 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-certs" pod "3a312903-03e0-43d4-8240-a8e60cdaedca" (UID: "3a312903-03e0-43d4-8240-a8e60cdaedca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:23:18.277141 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.277056 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-credential-socket" pod "3a312903-03e0-43d4-8240-a8e60cdaedca" (UID: "3a312903-03e0-43d4-8240-a8e60cdaedca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:23:18.277334 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.277301 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-socket" pod "3a312903-03e0-43d4-8240-a8e60cdaedca" (UID: "3a312903-03e0-43d4-8240-a8e60cdaedca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:23:18.282316 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.282195 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-podinfo" pod "3a312903-03e0-43d4-8240-a8e60cdaedca" (UID: "3a312903-03e0-43d4-8240-a8e60cdaedca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 02 07:23:18.282316 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.282211 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-kube-api-access-4869r" pod "3a312903-03e0-43d4-8240-a8e60cdaedca" (UID: "3a312903-03e0-43d4-8240-a8e60cdaedca"). InnerVolumeSpecName "kube-api-access-4869r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:23:18.282316 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.282284 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-token" pod "3a312903-03e0-43d4-8240-a8e60cdaedca" (UID: "3a312903-03e0-43d4-8240-a8e60cdaedca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 02 07:23:18.291198 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.291163 5025 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-envoy" pod "3a312903-03e0-43d4-8240-a8e60cdaedca" (UID: "3a312903-03e0-43d4-8240-a8e60cdaedca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 02 07:23:18.376322 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.376217 5025 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:18.376322 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.376270 5025 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-credential-socket\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:18.376322 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.376287 5025 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4869r\" (UniqueName: \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-kube-api-access-4869r\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:18.376322 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.376303 5025 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-workload-certs\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:18.376322 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.376319 5025 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-podinfo\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:18.376322 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.376332 5025 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a312903-03e0-43d4-8240-a8e60cdaedca-istiod-ca-cert\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:18.376799 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.376346 5025 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-data\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:18.376799 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.376359 5025 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-envoy\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:18.376799 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.376374 5025 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a312903-03e0-43d4-8240-a8e60cdaedca-istio-token\") on node \"ip-10-0-130-34.us-west-2.compute.internal\" DevicePath \"\"" Nov 02 07:23:18.552851 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.552810 5025 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp"] Nov 02 07:23:18.555439 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:18.555410 5025 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-c0d3ab-openshift-default-66d9c9c4cc-s5gmp"] Nov 02 07:23:19.681100 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:19.680918 5025 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" path="/var/lib/kubelet/pods/3a312903-03e0-43d4-8240-a8e60cdaedca/volumes" Nov 02 07:23:32.706138 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:23:32.706097 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:23:35.826712 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:35.826665 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:38.042953 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:38.042912 5025 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 02 07:23:38.043557 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:38.043529 5025 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-55f85cf5d7-4t2wh" podUID="c1cf776b-6c17-4791-9a7b-927b1bc044c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 02 07:23:47.012103 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:47.012054 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:23:47.196475 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:23:47.196440 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:02.855154 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:24:02.851245 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:24:32.705968 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:24:32.705922 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:24:36.153850 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:36.153816 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:24:39.360413 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:39.360378 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fbfc94f97-2dwj2_d5028bed-1e85-4b46-80c1-ca4d9b92d1c2/maas-api/0.log" Nov 02 07:24:40.629089 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:40.629047 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-fxx2z_a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f/postgres/0.log" Nov 02 07:24:44.773446 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:44.773366 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-twkd9_d5c2505e-f91e-4a17-a828-eab6162b785a/discovery/0.log" Nov 02 07:24:45.337558 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:45.337529 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-784594468b-pffjp_0cdac39f-8bc9-4943-98e8-173af92400bf/kube-auth-proxy/0.log" Nov 02 07:24:45.562493 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:45.562459 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-4t2wh_c1cf776b-6c17-4791-9a7b-927b1bc044c0/istio-proxy/0.log" Nov 02 07:24:46.263741 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:46.263713 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-8xsqm_2c57c5a3-8442-4aa2-967c-a9a80aa58d7b/istio-proxy/0.log" Nov 02 07:24:46.691684 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:46.691660 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-fmss9_24a7b3c3-fb09-4b70-aa73-858f52476799/istio-proxy/0.log" Nov 02 07:24:46.924043 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:46.924015 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-nsgcs_9b4f70ef-1722-4b66-9a44-5c771b7a24b9/istio-proxy/0.log" Nov 02 07:24:48.132321 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:48.132291 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9tcjh_81beb271-42bd-4b5e-999f-686a5020dee6/payload-pre-processing/0.log" Nov 02 07:24:48.359296 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:48.359265 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lfnlg_97699518-7b61-4b41-a574-ad41aefce208/payload-processing/0.log" Nov 02 07:24:49.445582 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:49.445552 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv_97ccb6cf-65c5-48e8-b65b-ad18ac981feb/main/0.log" Nov 02 07:24:49.720588 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:49.720508 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-btbz5_1fe9a903-1df8-4c6b-9a6f-1c5bb1168116/main/0.log" Nov 02 07:24:49.988263 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:49.988186 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-x888v_538b7b40-28f3-48e2-8172-f5198f6c63fb/main/0.log" Nov 02 07:24:50.211972 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:50.211937 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-m59z9_74ff53da-f5b3-482e-a31a-d01941e33562/main/0.log" Nov 02 07:24:50.445918 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:50.445887 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22_58816ff6-8725-4c1a-8b23-a9770d599670/main/0.log" Nov 02 07:24:50.904230 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:50.904148 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr_92fa156c-b466-447a-a715-33b3cafe2a35/main/0.log" Nov 02 07:24:51.128754 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:51.128725 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf_e1b38f9d-ab3b-4722-99b6-98ba6d9972f2/main/0.log" Nov 02 07:24:59.090401 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:24:59.090364 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:02.706650 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:25:02.706616 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:25:04.006514 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:04.006488 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fbfc94f97-2dwj2_d5028bed-1e85-4b46-80c1-ca4d9b92d1c2/maas-api/0.log" Nov 02 07:25:05.143261 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:05.143231 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-fxx2z_a16c03ab-56a9-4ab9-9cbc-6cfa18a1f25f/postgres/0.log" Nov 02 07:25:08.741349 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:08.741322 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-twkd9_d5c2505e-f91e-4a17-a828-eab6162b785a/discovery/0.log" Nov 02 07:25:09.333453 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:09.333420 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-784594468b-pffjp_0cdac39f-8bc9-4943-98e8-173af92400bf/kube-auth-proxy/0.log" Nov 02 07:25:09.558866 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:09.558823 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-4t2wh_c1cf776b-6c17-4791-9a7b-927b1bc044c0/istio-proxy/0.log" Nov 02 07:25:10.273256 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:10.273215 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-8xsqm_2c57c5a3-8442-4aa2-967c-a9a80aa58d7b/istio-proxy/0.log" Nov 02 07:25:10.713389 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:10.713361 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-fmss9_24a7b3c3-fb09-4b70-aa73-858f52476799/istio-proxy/0.log" Nov 02 07:25:10.940144 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:10.940110 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-55f85cf5d7-nsgcs_9b4f70ef-1722-4b66-9a44-5c771b7a24b9/istio-proxy/0.log" Nov 02 07:25:12.176590 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:12.176526 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9tcjh_81beb271-42bd-4b5e-999f-686a5020dee6/payload-pre-processing/0.log" Nov 02 07:25:12.390111 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:12.390062 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lfnlg_97699518-7b61-4b41-a574-ad41aefce208/payload-processing/0.log" Nov 02 07:25:13.284975 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:13.284929 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-htrsv_97ccb6cf-65c5-48e8-b65b-ad18ac981feb/main/0.log" Nov 02 07:25:13.547365 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:13.547273 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-btbz5_1fe9a903-1df8-4c6b-9a6f-1c5bb1168116/main/0.log" Nov 02 07:25:13.776290 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:13.776253 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-x888v_538b7b40-28f3-48e2-8172-f5198f6c63fb/main/0.log" Nov 02 07:25:13.999578 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:13.999547 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-m59z9_74ff53da-f5b3-482e-a31a-d01941e33562/main/0.log" Nov 02 07:25:14.221437 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:14.221396 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b86br22_58816ff6-8725-4c1a-8b23-a9770d599670/main/0.log" Nov 02 07:25:14.658664 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:14.658635 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-z9kqr_92fa156c-b466-447a-a715-33b3cafe2a35/main/0.log" Nov 02 07:25:14.872960 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:14.872924 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-t6ggf_e1b38f9d-ab3b-4722-99b6-98ba6d9972f2/main/0.log" Nov 02 07:25:32.707345 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:25:32.707308 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:25:36.478121 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:36.478069 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:25:45.232589 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:45.232553 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lfnlg_97699518-7b61-4b41-a574-ad41aefce208/payload-processing/0.log" Nov 02 07:25:45.450885 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:45.450858 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lfnlg_97699518-7b61-4b41-a574-ad41aefce208/payload-processing/0.log" Nov 02 07:25:48.087200 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:25:48.087157 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:02.706497 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:26:02.706458 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:26:25.720693 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:25.720661 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lfnlg_97699518-7b61-4b41-a574-ad41aefce208/payload-processing/0.log" Nov 02 07:26:25.945978 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:25.945950 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-lfnlg_97699518-7b61-4b41-a574-ad41aefce208/payload-processing/0.log" Nov 02 07:26:26.380629 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:26.380601 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:26.590640 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:26.590587 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:32.705965 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:26:32.705919 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor" Nov 02 07:26:34.800656 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:34.800625 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4vv5l_6ab6077a-6df9-4de9-8632-98830b9a8653/global-pull-secret-syncer/0.log" Nov 02 07:26:36.782851 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:36.782813 5025 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" limit=1024 Nov 02 07:26:37.698657 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:37.698626 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fc9xt_b6c85b19-43b7-4bb1-8bb3-0e1ff3006fad/konnectivity-agent/0.log" Nov 02 07:26:38.922265 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:38.922232 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-34.us-west-2.compute.internal_ceea1ff964927afa3d04bca3afe81300/haproxy/0.log" Nov 02 07:26:45.234215 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.234181 5025 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l"] Nov 02 07:26:45.234598 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.234572 5025 cpu_manager.go:414] "containerMap: removing container" podUID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerName="istio-proxy" Nov 02 07:26:45.234598 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.234584 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerName="istio-proxy" Nov 02 07:26:45.234671 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.234614 5025 cpu_manager.go:414] "containerMap: removing container" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerName="istio-proxy" Nov 02 07:26:45.234671 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.234622 5025 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerName="istio-proxy" Nov 02 07:26:45.234739 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.234684 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerName="istio-proxy" Nov 02 07:26:45.234739 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.234696 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerName="istio-proxy" Nov 02 07:26:45.234739 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.234704 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a312903-03e0-43d4-8240-a8e60cdaedca" containerName="istio-proxy" Nov 02 07:26:45.234739 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.234709 5025 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d7149a-760d-4214-9467-0e19bb2e98fe" containerName="istio-proxy" Nov 02 07:26:45.237697 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.237680 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.287499 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.287464 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-62jbh\"/\"kube-root-ca.crt\"" Nov 02 07:26:45.335140 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.335105 5025 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-62jbh\"/\"default-dockercfg-pqfzn\"" Nov 02 07:26:45.335341 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.335173 5025 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-62jbh\"/\"openshift-service-ca.crt\"" Nov 02 07:26:45.336765 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.336741 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l"] Nov 02 07:26:45.337282 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.337255 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw4tp\" (UniqueName: \"kubernetes.io/projected/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-kube-api-access-zw4tp\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.337356 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.337311 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-lib-modules\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.337494 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.337477 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-proc\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.337556 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.337521 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-podres\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.337665 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.337644 5025 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-sys\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.438718 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.438681 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-proc\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.438909 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.438731 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-podres\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.438909 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.438775 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-sys\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.438909 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.438806 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw4tp\" (UniqueName: \"kubernetes.io/projected/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-kube-api-access-zw4tp\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.438909 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.438808 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-proc\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.438909 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.438838 5025 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-lib-modules\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.438909 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.438885 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-sys\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.438909 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.438885 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-podres\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.439247 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.438984 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-lib-modules\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.495564 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.495484 5025 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw4tp\" (UniqueName: \"kubernetes.io/projected/11fc4803-f4ab-4c4f-b7d2-e973dfa128e1-kube-api-access-zw4tp\") pod \"perf-node-gather-daemonset-msj5l\" (UID: \"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1\") " pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.549440 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.549413 5025 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:45.911683 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:45.911655 5025 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l"] Nov 02 07:26:45.913191 ip-10-0-130-34 kubenswrapper[5025]: W1102 07:26:45.913160 5025 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod11fc4803_f4ab_4c4f_b7d2_e973dfa128e1.slice/crio-7dd7d324a0722bf6b8a559b5e61f46670fc4b34ad64f3ce667b08f8f1b6f1f4a WatchSource:0}: Error finding container 7dd7d324a0722bf6b8a559b5e61f46670fc4b34ad64f3ce667b08f8f1b6f1f4a: Status 404 returned error can't find the container with id 7dd7d324a0722bf6b8a559b5e61f46670fc4b34ad64f3ce667b08f8f1b6f1f4a Nov 02 07:26:46.190154 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:46.190117 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" event={"ID":"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1","Type":"ContainerStarted","Data":"eb65ae70aaf51f0f9e8f90a0bb7b870c52ae837e767397b10c59c551c5a7c8c5"} Nov 02 07:26:46.190341 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:46.190178 5025 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" event={"ID":"11fc4803-f4ab-4c4f-b7d2-e973dfa128e1","Type":"ContainerStarted","Data":"7dd7d324a0722bf6b8a559b5e61f46670fc4b34ad64f3ce667b08f8f1b6f1f4a"} Nov 02 07:26:46.190341 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:46.190231 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:46.299006 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:46.298953 5025 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" podStartSLOduration=1.29893985 podStartE2EDuration="1.29893985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 07:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 07:26:46.297560252 +0000 UTC m=+4182.745937007" watchObservedRunningTime="2026-11-02 07:26:46.29893985 +0000 UTC m=+4182.747316634" Nov 02 07:26:48.458873 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:48.458841 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c6b7817-19d9-45c8-a23b-f4f44e513890/alertmanager/0.log" Nov 02 07:26:48.857532 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:48.857447 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c6b7817-19d9-45c8-a23b-f4f44e513890/config-reloader/0.log" Nov 02 07:26:49.253526 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:49.253503 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c6b7817-19d9-45c8-a23b-f4f44e513890/kube-rbac-proxy-web/0.log" Nov 02 07:26:49.650265 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:49.650198 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c6b7817-19d9-45c8-a23b-f4f44e513890/kube-rbac-proxy/0.log" Nov 02 07:26:50.046023 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:50.045989 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c6b7817-19d9-45c8-a23b-f4f44e513890/kube-rbac-proxy-metric/0.log" Nov 02 07:26:50.443937 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:50.443898 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c6b7817-19d9-45c8-a23b-f4f44e513890/prom-label-proxy/0.log" Nov 02 07:26:50.843326 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:50.843249 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_7c6b7817-19d9-45c8-a23b-f4f44e513890/init-config-reloader/0.log" Nov 02 07:26:52.204031 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:52.204006 5025 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-62jbh/perf-node-gather-daemonset-msj5l" Nov 02 07:26:53.713766 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:53.713738 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wr46_0122975d-2172-4e2a-9af8-47f972bc5ecc/node-exporter/0.log" Nov 02 07:26:54.111323 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:54.111253 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wr46_0122975d-2172-4e2a-9af8-47f972bc5ecc/kube-rbac-proxy/0.log" Nov 02 07:26:54.535992 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:54.535966 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7wr46_0122975d-2172-4e2a-9af8-47f972bc5ecc/init-textfile/0.log" Nov 02 07:26:57.292443 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:57.292399 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p7227_720c25e0-1117-4a48-af60-9022daed412a/kube-rbac-proxy-main/0.log" Nov 02 07:26:57.692254 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:57.692223 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p7227_720c25e0-1117-4a48-af60-9022daed412a/kube-rbac-proxy-self/0.log" Nov 02 07:26:58.087880 ip-10-0-130-34 kubenswrapper[5025]: I1102 07:26:58.087801 5025 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-p7227_720c25e0-1117-4a48-af60-9022daed412a/openshift-state-metrics/0.log" Nov 02 07:27:02.707146 ip-10-0-130-34 kubenswrapper[5025]: E1102 07:27:02.707105 5025 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-x48kz" containerName="extractor"